2024-03-24T09:27:11.824 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-24T09:27:11.824 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --name pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi --archive /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619043 --owner scheduled_pdonnell@teuthology --verbose -- /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619043/orig.config.yaml 2024-03-24T09:27:11.875 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T09:27:11.979 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619043 branch: wip-batrick-testing-20240323.003144-squid description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: pdonnell@redhat.com first_in_suite: false job_id: '7619043' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/reef ' name: pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240323.003144-squid ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 21b3bd672163092b4e542f772797fcba02d3d4de ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 21b3bd672163092b4e542f772797fcba02d3d4de kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240323.003144-squid sha1: d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 4284 sha1: 21b3bd672163092b4e542f772797fcba02d3d4de sleep_before_teardown: 0 subset: 405/512 suite: fs suite_branch: wip-batrick-testing-20240323.003144-squid suite_path: /home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph suite_sha1: d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b targets: smithi022.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEZMU8wfzxXYjkToy1O0HCIJPcSvIjN9CZeeueWYCafZ3yHXFp7VdpukJH+++twxJdbF03+oDj2xsVwLpKYyyH4= smithi155.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEWIBKXao9l9HY/Ss+f6KvxgkCPZm0HzzlnoKyUtczCgpOtgmJicC3gx77Xry9WD0tK7rh+bATRAIlFjgHm5f/s= tasks: - install: branch: reef exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: compiled_cephadm_branch: reef conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:reef roleless: true - print: '**** done end installing reef cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-24_04:56:01 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-03-24T09:27:11.979 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa; will attempt to use it 2024-03-24T09:27:11.980 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks 2024-03-24T09:27:11.980 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-24T09:27:11.981 INFO:teuthology.task.internal:Checking packages... 2024-03-24T09:27:12.007 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '21b3bd672163092b4e542f772797fcba02d3d4de' 2024-03-24T09:27:12.008 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T09:27:12.008 INFO:teuthology.packaging:ref: None 2024-03-24T09:27:12.008 INFO:teuthology.packaging:tag: None 2024-03-24T09:27:12.008 INFO:teuthology.packaging:branch: wip-batrick-testing-20240323.003144-squid 2024-03-24T09:27:12.008 INFO:teuthology.packaging:sha1: 21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T09:27:12.008 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240323.003144-squid 2024-03-24T09:27:12.148 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1855.g21b3bd67 2024-03-24T09:27:12.149 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-24T09:27:12.155 INFO:teuthology.task.internal:no buildpackages task found 2024-03-24T09:27:12.156 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-24T09:27:12.175 INFO:teuthology.task.internal:Saving configuration 2024-03-24T09:27:12.193 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-24T09:27:12.199 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T09:27:12.221 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi022.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619043', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 09:21:57.903781', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEZMU8wfzxXYjkToy1O0HCIJPcSvIjN9CZeeueWYCafZ3yHXFp7VdpukJH+++twxJdbF03+oDj2xsVwLpKYyyH4='} 2024-03-24T09:27:12.243 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi155.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619043', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 09:21:57.902602', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEWIBKXao9l9HY/Ss+f6KvxgkCPZm0HzzlnoKyUtczCgpOtgmJicC3gx77Xry9WD0tK7rh+bATRAIlFjgHm5f/s='} 2024-03-24T09:27:12.244 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-24T09:27:12.250 INFO:teuthology.task.internal:roles: ubuntu@smithi022.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-03-24T09:27:12.250 INFO:teuthology.task.internal:roles: ubuntu@smithi155.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-03-24T09:27:12.250 INFO:teuthology.run_tasks:Running task console_log... 2024-03-24T09:27:12.320 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7e8dc41160>, signals=[15]) 2024-03-24T09:27:12.320 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-24T09:27:12.326 INFO:teuthology.task.internal:Opening connections... 2024-03-24T09:27:12.326 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi022.front.sepia.ceph.com 2024-03-24T09:27:12.327 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T09:27:12.429 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi155.front.sepia.ceph.com 2024-03-24T09:27:12.430 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T09:27:12.516 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-24T09:27:12.523 DEBUG:teuthology.orchestra.run.smithi022:> uname -m 2024-03-24T09:27:12.540 INFO:teuthology.orchestra.run.smithi022.stdout:x86_64 2024-03-24T09:27:12.540 DEBUG:teuthology.orchestra.run.smithi022:> cat /etc/os-release 2024-03-24T09:27:12.597 INFO:teuthology.orchestra.run.smithi022.stdout:NAME="CentOS Stream" 2024-03-24T09:27:12.597 INFO:teuthology.orchestra.run.smithi022.stdout:VERSION="9" 2024-03-24T09:27:12.597 INFO:teuthology.orchestra.run.smithi022.stdout:ID="centos" 2024-03-24T09:27:12.597 INFO:teuthology.orchestra.run.smithi022.stdout:ID_LIKE="rhel fedora" 2024-03-24T09:27:12.597 INFO:teuthology.orchestra.run.smithi022.stdout:VERSION_ID="9" 2024-03-24T09:27:12.597 INFO:teuthology.orchestra.run.smithi022.stdout:PLATFORM_ID="platform:el9" 2024-03-24T09:27:12.597 INFO:teuthology.orchestra.run.smithi022.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T09:27:12.597 INFO:teuthology.orchestra.run.smithi022.stdout:ANSI_COLOR="0;31" 2024-03-24T09:27:12.597 INFO:teuthology.orchestra.run.smithi022.stdout:LOGO="fedora-logo-icon" 2024-03-24T09:27:12.597 INFO:teuthology.orchestra.run.smithi022.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T09:27:12.597 INFO:teuthology.orchestra.run.smithi022.stdout:HOME_URL="https://centos.org/" 2024-03-24T09:27:12.597 INFO:teuthology.orchestra.run.smithi022.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T09:27:12.598 INFO:teuthology.orchestra.run.smithi022.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T09:27:12.598 INFO:teuthology.orchestra.run.smithi022.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T09:27:12.598 INFO:teuthology.lock.ops:Updating smithi022.front.sepia.ceph.com on lock server 2024-03-24T09:27:12.621 DEBUG:teuthology.orchestra.run.smithi155:> uname -m 2024-03-24T09:27:12.638 INFO:teuthology.orchestra.run.smithi155.stdout:x86_64 2024-03-24T09:27:12.639 DEBUG:teuthology.orchestra.run.smithi155:> cat /etc/os-release 2024-03-24T09:27:12.693 INFO:teuthology.orchestra.run.smithi155.stdout:NAME="CentOS Stream" 2024-03-24T09:27:12.693 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION="9" 2024-03-24T09:27:12.693 INFO:teuthology.orchestra.run.smithi155.stdout:ID="centos" 2024-03-24T09:27:12.693 INFO:teuthology.orchestra.run.smithi155.stdout:ID_LIKE="rhel fedora" 2024-03-24T09:27:12.694 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION_ID="9" 2024-03-24T09:27:12.694 INFO:teuthology.orchestra.run.smithi155.stdout:PLATFORM_ID="platform:el9" 2024-03-24T09:27:12.694 INFO:teuthology.orchestra.run.smithi155.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T09:27:12.694 INFO:teuthology.orchestra.run.smithi155.stdout:ANSI_COLOR="0;31" 2024-03-24T09:27:12.694 INFO:teuthology.orchestra.run.smithi155.stdout:LOGO="fedora-logo-icon" 2024-03-24T09:27:12.694 INFO:teuthology.orchestra.run.smithi155.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T09:27:12.694 INFO:teuthology.orchestra.run.smithi155.stdout:HOME_URL="https://centos.org/" 2024-03-24T09:27:12.694 INFO:teuthology.orchestra.run.smithi155.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T09:27:12.694 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T09:27:12.694 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T09:27:12.694 INFO:teuthology.lock.ops:Updating smithi155.front.sepia.ceph.com on lock server 2024-03-24T09:27:12.712 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-24T09:27:12.720 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-24T09:27:12.725 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-24T09:27:12.725 DEBUG:teuthology.orchestra.run.smithi022:> test '!' -e /home/ubuntu/cephtest 2024-03-24T09:27:12.728 DEBUG:teuthology.orchestra.run.smithi155:> test '!' -e /home/ubuntu/cephtest 2024-03-24T09:27:12.748 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-24T09:27:12.754 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-24T09:27:12.754 DEBUG:teuthology.orchestra.run.smithi022:> test -z $(ls -A /var/lib/ceph) 2024-03-24T09:27:12.785 DEBUG:teuthology.orchestra.run.smithi155:> test -z $(ls -A /var/lib/ceph) 2024-03-24T09:27:12.881 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-24T09:27:12.936 INFO:teuthology.run_tasks:Running task kernel... 2024-03-24T09:27:12.949 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-24T09:27:12.949 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-24T09:27:12.950 DEBUG:teuthology.orchestra.run.smithi022:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T09:27:12.950 DEBUG:teuthology.orchestra.run.smithi155:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T09:27:12.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T09:27:12.967 DEBUG:teuthology.orchestra.run.smithi155:> uname -r 2024-03-24T09:27:12.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T09:27:12.968 DEBUG:teuthology.orchestra.run.smithi022:> uname -r 2024-03-24T09:27:13.021 INFO:teuthology.orchestra.run.smithi155.stdout:5.14.0-361.el9.x86_64 2024-03-24T09:27:13.022 INFO:teuthology.task.kernel:Running kernel on smithi155: 5.14.0-361.el9.x86_64 2024-03-24T09:27:13.022 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum install -y kernel 2024-03-24T09:27:13.026 INFO:teuthology.orchestra.run.smithi022.stdout:5.14.0-361.el9.x86_64 2024-03-24T09:27:13.026 INFO:teuthology.task.kernel:Running kernel on smithi022: 5.14.0-361.el9.x86_64 2024-03-24T09:27:13.026 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum install -y kernel 2024-03-24T09:27:15.281 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-24T09:27:15.357 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS-9 - AppStream 9.6 MB/s | 6.7 MB 00:00 2024-03-24T09:27:16.765 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS-9 - BaseOS 5.9 MB/s | 2.0 MB 00:00 2024-03-24T09:27:17.015 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2024-03-24T09:27:18.089 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - BaseOS 8.2 MB/s | 8.0 MB 00:00 2024-03-24T09:27:18.423 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - BaseOS 7.5 MB/s | 8.0 MB 00:01 2024-03-24T09:27:21.306 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - AppStream 8.9 MB/s | 19 MB 00:02 2024-03-24T09:27:21.657 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - AppStream 8.6 MB/s | 19 MB 00:02 2024-03-24T09:27:25.382 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-24T09:27:25.751 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-03-24T09:27:26.757 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - Extras packages 84 kB/s | 16 kB 00:00 2024-03-24T09:27:27.392 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-03-24T09:27:28.736 INFO:teuthology.orchestra.run.smithi022.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-24T09:27:33.801 INFO:teuthology.orchestra.run.smithi022.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-24T09:27:35.982 INFO:teuthology.orchestra.run.smithi022.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-24T09:27:35.983 INFO:teuthology.orchestra.run.smithi022.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-24T09:27:36.085 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-24T09:27:36.097 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T09:27:36.097 INFO:teuthology.orchestra.run.smithi022.stdout: Package Architecture Version Repository Size 2024-03-24T09:27:36.097 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T09:27:36.098 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-24T09:27:36.098 INFO:teuthology.orchestra.run.smithi022.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-24T09:27:36.098 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-24T09:27:36.098 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-24T09:27:36.098 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-24T09:27:36.098 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-24T09:27:36.098 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:27:36.098 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-24T09:27:36.098 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T09:27:36.098 INFO:teuthology.orchestra.run.smithi022.stdout:Install 4 Packages 2024-03-24T09:27:36.098 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:27:36.098 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 76 M 2024-03-24T09:27:36.099 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 123 M 2024-03-24T09:27:36.099 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-24T09:27:36.269 INFO:teuthology.orchestra.run.smithi022.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 432 kB/s | 44 kB 00:00 2024-03-24T09:27:36.644 INFO:teuthology.orchestra.run.smithi022.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 31 MB/s | 15 MB 00:00 2024-03-24T09:27:37.027 INFO:teuthology.orchestra.run.smithi022.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 36 MB/s | 27 MB 00:00 2024-03-24T09:27:37.353 INFO:teuthology.orchestra.run.smithi022.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 28 MB/s | 33 MB 00:01 2024-03-24T09:27:37.353 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:27:37.353 INFO:teuthology.orchestra.run.smithi022.stdout:Total 60 MB/s | 76 MB 00:01 2024-03-24T09:27:38.049 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-24T09:27:38.500 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-24T09:27:38.500 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-24T09:27:39.537 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-24T09:27:39.537 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-24T09:27:42.166 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-24T09:27:42.799 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T09:27:42.887 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T09:27:45.717 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T09:27:45.856 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T09:27:47.776 INFO:teuthology.orchestra.run.smithi155.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 21 MB 00:20 2024-03-24T09:27:50.930 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T09:27:50.975 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T09:27:52.815 INFO:teuthology.orchestra.run.smithi155.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-24T09:27:54.963 INFO:teuthology.orchestra.run.smithi155.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-24T09:27:54.964 INFO:teuthology.orchestra.run.smithi155.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-24T09:27:55.057 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-24T09:27:55.070 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-24T09:27:55.070 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2024-03-24T09:27:55.070 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-24T09:27:55.070 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-24T09:27:55.070 INFO:teuthology.orchestra.run.smithi155.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-24T09:27:55.070 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-24T09:27:55.071 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-24T09:27:55.071 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-24T09:27:55.071 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-24T09:27:55.071 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:27:55.071 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-24T09:27:55.071 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-24T09:27:55.071 INFO:teuthology.orchestra.run.smithi155.stdout:Install 4 Packages 2024-03-24T09:27:55.071 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:27:55.072 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 76 M 2024-03-24T09:27:55.072 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 123 M 2024-03-24T09:27:55.072 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-24T09:27:55.883 INFO:teuthology.orchestra.run.smithi155.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 66 kB/s | 44 kB 00:00 2024-03-24T09:27:56.080 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T09:28:14.955 INFO:teuthology.orchestra.run.smithi155.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 773 kB/s | 15 MB 00:19 2024-03-24T09:28:26.793 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T09:28:26.794 INFO:teuthology.orchestra.run.smithi022.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T09:28:26.794 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:28:26.839 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T09:28:27.262 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T09:28:27.263 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T09:28:27.263 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T09:28:27.263 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T09:28:27.703 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T09:28:27.703 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:28:27.703 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-24T09:28:27.703 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T09:28:27.704 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T09:28:27.704 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T09:28:27.704 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T09:28:27.704 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:28:27.704 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-24T09:28:28.051 DEBUG:teuthology.orchestra.run.smithi022:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T09:28:28.152 INFO:teuthology.orchestra.run.smithi022.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T09:28:28.152 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T09:28:28.152 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T09:28:28.152 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T09:28:28.152 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T09:28:28.153 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-03-24T09:28:28.153 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi022.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T09:28:28.153 DEBUG:teuthology.orchestra.run.smithi022:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T09:28:28.235 INFO:teuthology.orchestra.run.smithi022.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T09:28:28.235 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T09:28:28.235 DEBUG:teuthology.orchestra.run.smithi022:> sudo rpm -qi grub2-tools 2024-03-24T09:28:28.282 INFO:teuthology.orchestra.run.smithi022.stdout:Name : grub2-tools 2024-03-24T09:28:28.282 INFO:teuthology.orchestra.run.smithi022.stdout:Epoch : 1 2024-03-24T09:28:28.282 INFO:teuthology.orchestra.run.smithi022.stdout:Version : 2.06 2024-03-24T09:28:28.282 INFO:teuthology.orchestra.run.smithi022.stdout:Release : 68.el9 2024-03-24T09:28:28.282 INFO:teuthology.orchestra.run.smithi022.stdout:Architecture: x86_64 2024-03-24T09:28:28.282 INFO:teuthology.orchestra.run.smithi022.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T09:28:28.282 INFO:teuthology.orchestra.run.smithi022.stdout:Group : Unspecified 2024-03-24T09:28:28.282 INFO:teuthology.orchestra.run.smithi022.stdout:Size : 8264254 2024-03-24T09:28:28.282 INFO:teuthology.orchestra.run.smithi022.stdout:License : GPLv3+ 2024-03-24T09:28:28.283 INFO:teuthology.orchestra.run.smithi022.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T09:28:28.283 INFO:teuthology.orchestra.run.smithi022.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T09:28:28.283 INFO:teuthology.orchestra.run.smithi022.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T09:28:28.283 INFO:teuthology.orchestra.run.smithi022.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T09:28:28.283 INFO:teuthology.orchestra.run.smithi022.stdout:Packager : builder@centos.org 2024-03-24T09:28:28.283 INFO:teuthology.orchestra.run.smithi022.stdout:Vendor : CentOS 2024-03-24T09:28:28.283 INFO:teuthology.orchestra.run.smithi022.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T09:28:28.283 INFO:teuthology.orchestra.run.smithi022.stdout:Summary : Support tools for GRUB. 2024-03-24T09:28:28.283 INFO:teuthology.orchestra.run.smithi022.stdout:Description : 2024-03-24T09:28:28.283 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:28:28.283 INFO:teuthology.orchestra.run.smithi022.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T09:28:28.283 INFO:teuthology.orchestra.run.smithi022.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T09:28:28.284 INFO:teuthology.orchestra.run.smithi022.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T09:28:28.284 INFO:teuthology.orchestra.run.smithi022.stdout:hardware devices. 2024-03-24T09:28:28.284 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:28:28.284 INFO:teuthology.orchestra.run.smithi022.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T09:28:28.285 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T09:28:28.285 INFO:teuthology.task.kernel:Updating grub on smithi022 to boot 5.14.0-430.el9.x86_64 2024-03-24T09:28:28.285 DEBUG:teuthology.orchestra.run.smithi022:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T09:28:28.870 INFO:teuthology.orchestra.run.smithi022.stderr:Generating grub configuration file ... 2024-03-24T09:28:30.233 INFO:teuthology.orchestra.run.smithi022.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T09:28:30.264 INFO:teuthology.orchestra.run.smithi022.stderr:done 2024-03-24T09:28:30.267 DEBUG:teuthology.orchestra.run.smithi022:> mktemp 2024-03-24T09:28:30.283 INFO:teuthology.orchestra.run.smithi022.stdout:/tmp/tmp.iGBfoVwcVB 2024-03-24T09:28:30.283 DEBUG:teuthology.orchestra.run.smithi022:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.iGBfoVwcVB 2024-03-24T09:28:30.356 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 0666 /tmp/tmp.iGBfoVwcVB 2024-03-24T09:28:30.504 DEBUG:teuthology.orchestra.remote:smithi022:/tmp/tmp.iGBfoVwcVB is 6KB 2024-03-24T09:28:30.555 DEBUG:teuthology.orchestra.run.smithi022:> rm -fr /tmp/tmp.iGBfoVwcVB 2024-03-24T09:28:30.570 DEBUG:teuthology.orchestra.run.smithi022:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T09:28:30.639 INFO:teuthology.orchestra.run.smithi022.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T09:28:30.639 INFO:teuthology.orchestra.run.smithi022.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T09:28:30.639 INFO:teuthology.orchestra.run.smithi022.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T09:28:30.639 INFO:teuthology.orchestra.run.smithi022.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T09:28:30.640 DEBUG:teuthology.orchestra.run.smithi022:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T09:28:30.770 INFO:teuthology.orchestra.run.smithi155.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 806 kB/s | 27 MB 00:34 2024-03-24T09:28:30.814 DEBUG:teuthology.orchestra.run.smithi022:> sudo shutdown -r now 2024-03-24T09:28:37.278 INFO:teuthology.orchestra.run.smithi155.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 814 kB/s | 33 MB 00:42 2024-03-24T09:28:37.279 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:28:37.279 INFO:teuthology.orchestra.run.smithi155.stdout:Total 1.8 MB/s | 76 MB 00:42 2024-03-24T09:28:37.935 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-24T09:28:38.300 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-24T09:28:38.301 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-24T09:28:39.363 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-24T09:28:39.364 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-24T09:28:41.848 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-24T09:28:42.456 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T09:28:42.544 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T09:28:45.289 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T09:28:45.427 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T09:28:50.121 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T09:28:50.245 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T09:28:54.956 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T09:29:00.818 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-03-24T09:29:00.822 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi022.front.sepia.ceph.com' 2024-03-24T09:29:00.824 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T09:29:24.130 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T09:29:24.130 INFO:teuthology.orchestra.run.smithi155.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T09:29:24.130 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:29:24.172 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T09:29:24.577 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T09:29:24.577 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T09:29:24.577 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T09:29:24.577 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T09:29:25.002 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T09:29:25.002 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:29:25.002 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-24T09:29:25.002 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T09:29:25.002 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T09:29:25.002 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T09:29:25.003 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T09:29:25.003 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:29:25.003 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-24T09:29:25.329 DEBUG:teuthology.orchestra.run.smithi155:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T09:29:25.438 INFO:teuthology.orchestra.run.smithi155.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T09:29:25.439 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T09:29:25.439 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T09:29:25.439 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T09:29:25.439 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T09:29:25.439 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-03-24T09:29:25.439 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi155.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T09:29:25.439 DEBUG:teuthology.orchestra.run.smithi155:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T09:29:25.518 INFO:teuthology.orchestra.run.smithi155.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T09:29:25.519 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T09:29:25.519 DEBUG:teuthology.orchestra.run.smithi155:> sudo rpm -qi grub2-tools 2024-03-24T09:29:25.552 INFO:teuthology.orchestra.run.smithi155.stdout:Name : grub2-tools 2024-03-24T09:29:25.552 INFO:teuthology.orchestra.run.smithi155.stdout:Epoch : 1 2024-03-24T09:29:25.552 INFO:teuthology.orchestra.run.smithi155.stdout:Version : 2.06 2024-03-24T09:29:25.552 INFO:teuthology.orchestra.run.smithi155.stdout:Release : 68.el9 2024-03-24T09:29:25.552 INFO:teuthology.orchestra.run.smithi155.stdout:Architecture: x86_64 2024-03-24T09:29:25.552 INFO:teuthology.orchestra.run.smithi155.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T09:29:25.552 INFO:teuthology.orchestra.run.smithi155.stdout:Group : Unspecified 2024-03-24T09:29:25.552 INFO:teuthology.orchestra.run.smithi155.stdout:Size : 8264254 2024-03-24T09:29:25.552 INFO:teuthology.orchestra.run.smithi155.stdout:License : GPLv3+ 2024-03-24T09:29:25.553 INFO:teuthology.orchestra.run.smithi155.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T09:29:25.553 INFO:teuthology.orchestra.run.smithi155.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T09:29:25.553 INFO:teuthology.orchestra.run.smithi155.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T09:29:25.553 INFO:teuthology.orchestra.run.smithi155.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T09:29:25.553 INFO:teuthology.orchestra.run.smithi155.stdout:Packager : builder@centos.org 2024-03-24T09:29:25.553 INFO:teuthology.orchestra.run.smithi155.stdout:Vendor : CentOS 2024-03-24T09:29:25.553 INFO:teuthology.orchestra.run.smithi155.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T09:29:25.553 INFO:teuthology.orchestra.run.smithi155.stdout:Summary : Support tools for GRUB. 2024-03-24T09:29:25.553 INFO:teuthology.orchestra.run.smithi155.stdout:Description : 2024-03-24T09:29:25.553 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:29:25.553 INFO:teuthology.orchestra.run.smithi155.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T09:29:25.553 INFO:teuthology.orchestra.run.smithi155.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T09:29:25.553 INFO:teuthology.orchestra.run.smithi155.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T09:29:25.553 INFO:teuthology.orchestra.run.smithi155.stdout:hardware devices. 2024-03-24T09:29:25.553 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:29:25.553 INFO:teuthology.orchestra.run.smithi155.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T09:29:25.554 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T09:29:25.554 INFO:teuthology.task.kernel:Updating grub on smithi155 to boot 5.14.0-430.el9.x86_64 2024-03-24T09:29:25.554 DEBUG:teuthology.orchestra.run.smithi155:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T09:29:26.133 INFO:teuthology.orchestra.run.smithi155.stderr:Generating grub configuration file ... 2024-03-24T09:29:27.553 INFO:teuthology.orchestra.run.smithi155.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T09:29:27.591 INFO:teuthology.orchestra.run.smithi155.stderr:done 2024-03-24T09:29:27.594 DEBUG:teuthology.orchestra.run.smithi155:> mktemp 2024-03-24T09:29:27.609 INFO:teuthology.orchestra.run.smithi155.stdout:/tmp/tmp.y2bBkeaiFH 2024-03-24T09:29:27.610 DEBUG:teuthology.orchestra.run.smithi155:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.y2bBkeaiFH 2024-03-24T09:29:27.679 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0666 /tmp/tmp.y2bBkeaiFH 2024-03-24T09:29:27.819 DEBUG:teuthology.orchestra.remote:smithi155:/tmp/tmp.y2bBkeaiFH is 6KB 2024-03-24T09:29:27.868 DEBUG:teuthology.orchestra.run.smithi155:> rm -fr /tmp/tmp.y2bBkeaiFH 2024-03-24T09:29:27.882 DEBUG:teuthology.orchestra.run.smithi155:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T09:29:27.947 INFO:teuthology.orchestra.run.smithi155.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T09:29:27.947 INFO:teuthology.orchestra.run.smithi155.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T09:29:27.947 INFO:teuthology.orchestra.run.smithi155.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T09:29:27.947 INFO:teuthology.orchestra.run.smithi155.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T09:29:27.949 DEBUG:teuthology.orchestra.run.smithi155:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T09:29:28.099 DEBUG:teuthology.orchestra.run.smithi155:> sudo shutdown -r now 2024-03-24T09:29:58.104 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-03-24T09:29:58.104 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T09:29:58.105 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T09:30:00.830 DEBUG:teuthology.orchestra.remote:timed out 2024-03-24T09:30:09.839 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi022.front.sepia.ceph.com' 2024-03-24T09:30:09.840 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T09:30:10.139 DEBUG:teuthology.orchestra.run.smithi022:> true 2024-03-24T09:30:10.697 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi022.front.sepia.ceph.com' 2024-03-24T09:30:10.698 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-430.el9.x86_64"... 2024-03-24T09:30:10.698 DEBUG:teuthology.orchestra.run.smithi022:> uname -r 2024-03-24T09:30:10.719 INFO:teuthology.orchestra.run.smithi022.stdout:5.14.0-430.el9.x86_64 2024-03-24T09:30:10.719 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T09:30:10.720 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T09:30:10.720 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T09:30:11.721 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-03-24T09:30:11.721 DEBUG:teuthology.orchestra.run.smithi022:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T09:30:11.779 INFO:teuthology.orchestra.run.smithi022.stdout:ttyS1 2024-03-24T09:30:11.830 DEBUG:teuthology.parallel:result is None 2024-03-24T09:30:16.478 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T09:30:25.480 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T09:30:25.481 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T09:30:28.542 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T09:30:40.544 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T09:30:40.544 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T09:31:40.547 DEBUG:teuthology.orchestra.remote:timed out 2024-03-24T09:31:55.548 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T09:31:55.549 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T09:31:55.996 DEBUG:teuthology.orchestra.run.smithi155:> true 2024-03-24T09:31:56.399 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T09:31:56.399 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-430.el9.x86_64"... 2024-03-24T09:31:56.400 DEBUG:teuthology.orchestra.run.smithi155:> uname -r 2024-03-24T09:31:56.414 INFO:teuthology.orchestra.run.smithi155.stdout:5.14.0-430.el9.x86_64 2024-03-24T09:31:56.415 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T09:31:56.415 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T09:31:56.415 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T09:31:57.416 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-03-24T09:31:57.416 DEBUG:teuthology.orchestra.run.smithi155:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T09:31:57.492 INFO:teuthology.orchestra.run.smithi155.stdout:ttyS1 2024-03-24T09:31:57.521 DEBUG:teuthology.parallel:result is None 2024-03-24T09:31:57.521 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-24T09:31:57.528 INFO:teuthology.task.internal:Creating test directory... 2024-03-24T09:31:57.529 DEBUG:teuthology.orchestra.run.smithi022:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T09:31:57.531 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T09:31:57.549 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-24T09:31:57.558 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-24T09:31:57.568 INFO:teuthology.task.internal:Creating archive directory... 2024-03-24T09:31:57.568 DEBUG:teuthology.orchestra.run.smithi022:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T09:31:57.589 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T09:31:57.621 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-24T09:31:57.626 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-24T09:31:57.626 DEBUG:teuthology.orchestra.run.smithi022:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T09:31:57.651 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T09:31:57.689 INFO:teuthology.orchestra.run.smithi022.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T09:31:57.690 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T09:31:57.704 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T09:31:57.704 INFO:teuthology.orchestra.run.smithi022.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T09:31:57.706 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-24T09:31:57.712 INFO:teuthology.task.internal:Configuring sudo... 2024-03-24T09:31:57.712 DEBUG:teuthology.orchestra.run.smithi022:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T09:31:57.749 DEBUG:teuthology.orchestra.run.smithi155:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T09:31:57.778 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-24T09:31:57.786 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-24T09:31:57.786 DEBUG:teuthology.orchestra.run.smithi022:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T09:31:57.820 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T09:31:57.835 DEBUG:teuthology.orchestra.run.smithi022:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T09:31:57.917 DEBUG:teuthology.orchestra.run.smithi022:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T09:31:57.965 DEBUG:teuthology.orchestra.run.smithi022:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T09:31:58.043 DEBUG:teuthology.orchestra.run.smithi022:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T09:31:58.090 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T09:31:58.090 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T09:31:58.158 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T09:31:58.196 DEBUG:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T09:31:58.257 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T09:31:58.337 DEBUG:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T09:31:58.384 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-24T09:31:58.384 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T09:31:58.451 DEBUG:teuthology.orchestra.run.smithi022:> sudo service rsyslog restart 2024-03-24T09:31:58.453 DEBUG:teuthology.orchestra.run.smithi155:> sudo service rsyslog restart 2024-03-24T09:31:58.502 INFO:teuthology.orchestra.run.smithi022.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T09:31:58.533 INFO:teuthology.orchestra.run.smithi155.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T09:31:59.009 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-24T09:31:59.016 INFO:teuthology.task.internal:Starting timer... 2024-03-24T09:31:59.016 INFO:teuthology.run_tasks:Running task pcp... 2024-03-24T09:31:59.025 INFO:teuthology.run_tasks:Running task selinux... 2024-03-24T09:31:59.033 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-24T09:31:59.033 DEBUG:teuthology.orchestra.run.smithi022:> sudo service auditd rotate 2024-03-24T09:31:59.106 INFO:teuthology.orchestra.run.smithi022.stdout:Rotating logs: 2024-03-24T09:31:59.108 DEBUG:teuthology.orchestra.run.smithi155:> sudo service auditd rotate 2024-03-24T09:31:59.161 INFO:teuthology.orchestra.run.smithi155.stdout:Rotating logs: 2024-03-24T09:31:59.162 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-24T09:31:59.163 DEBUG:teuthology.orchestra.run.smithi022:> /usr/sbin/getenforce 2024-03-24T09:31:59.183 INFO:teuthology.orchestra.run.smithi022.stdout:Permissive 2024-03-24T09:31:59.183 DEBUG:teuthology.orchestra.run.smithi155:> /usr/sbin/getenforce 2024-03-24T09:31:59.206 INFO:teuthology.orchestra.run.smithi155.stdout:Permissive 2024-03-24T09:31:59.206 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi022.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi155.front.sepia.ceph.com': 'permissive'} 2024-03-24T09:31:59.206 DEBUG:teuthology.orchestra.run.smithi022:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T09:31:59.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T09:31:59.252 DEBUG:teuthology.orchestra.run.smithi155:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T09:31:59.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T09:31:59.280 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-24T09:31:59.280 DEBUG:teuthology.orchestra.run.smithi022:> sudo /usr/sbin/setenforce permissive 2024-03-24T09:31:59.325 DEBUG:teuthology.orchestra.run.smithi155:> sudo /usr/sbin/setenforce permissive 2024-03-24T09:31:59.357 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-24T09:31:59.365 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-24T09:31:59.371 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-24T09:31:59.509 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-24T09:31:59.523 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-24T09:31:59.525 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi022.front.sepia.ceph.com,smithi155.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-24T09:36:05.108 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi022.front.sepia.ceph.com'), Remote(name='ubuntu@smithi155.front.sepia.ceph.com')] 2024-03-24T09:36:05.109 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi022.front.sepia.ceph.com' 2024-03-24T09:36:05.110 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T09:36:05.195 DEBUG:teuthology.orchestra.run.smithi022:> true 2024-03-24T09:36:05.273 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi022.front.sepia.ceph.com' 2024-03-24T09:36:05.273 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T09:36:05.273 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T09:36:05.360 DEBUG:teuthology.orchestra.run.smithi155:> true 2024-03-24T09:36:05.445 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T09:36:05.446 INFO:teuthology.run_tasks:Running task clock... 2024-03-24T09:36:05.456 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-24T09:36:05.456 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T09:36:05.457 DEBUG:teuthology.orchestra.run.smithi022:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T09:36:05.461 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T09:36:05.461 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T09:36:05.489 INFO:teuthology.orchestra.run.smithi022.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T09:36:05.506 INFO:teuthology.orchestra.run.smithi022.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T09:36:05.516 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T09:36:05.533 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T09:36:05.536 INFO:teuthology.orchestra.run.smithi022.stderr:sudo: ntpd: command not found 2024-03-24T09:36:05.549 INFO:teuthology.orchestra.run.smithi022.stdout:506 Cannot talk to daemon 2024-03-24T09:36:05.562 INFO:teuthology.orchestra.run.smithi155.stderr:sudo: ntpd: command not found 2024-03-24T09:36:05.566 INFO:teuthology.orchestra.run.smithi022.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T09:36:05.576 INFO:teuthology.orchestra.run.smithi155.stdout:506 Cannot talk to daemon 2024-03-24T09:36:05.583 INFO:teuthology.orchestra.run.smithi022.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T09:36:05.592 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T09:36:05.608 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T09:36:05.629 INFO:teuthology.orchestra.run.smithi022.stderr:bash: line 1: ntpq: command not found 2024-03-24T09:36:05.634 INFO:teuthology.orchestra.run.smithi022.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T09:36:05.634 INFO:teuthology.orchestra.run.smithi022.stdout:=============================================================================== 2024-03-24T09:36:05.634 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T09:36:05.634 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T09:36:05.634 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T09:36:05.634 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T09:36:05.658 INFO:teuthology.orchestra.run.smithi155.stderr:bash: line 1: ntpq: command not found 2024-03-24T09:36:05.662 INFO:teuthology.orchestra.run.smithi155.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T09:36:05.663 INFO:teuthology.orchestra.run.smithi155.stdout:=============================================================================== 2024-03-24T09:36:05.663 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T09:36:05.663 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T09:36:05.663 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T09:36:05.663 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T09:36:05.663 INFO:teuthology.run_tasks:Running task install... 2024-03-24T09:36:05.671 DEBUG:teuthology.task.install:project ceph 2024-03-24T09:36:05.671 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de'}} 2024-03-24T09:36:05.671 DEBUG:teuthology.task.install:config {'branch': 'reef', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de'} 2024-03-24T09:36:05.671 INFO:teuthology.task.install:Using flavor: default 2024-03-24T09:36:05.677 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-24T09:36:05.677 INFO:teuthology.task.install:extra packages: [] 2024-03-24T09:36:05.677 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'reef', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de', 'tag': None, 'wait_for_package': False} 2024-03-24T09:36:05.678 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T09:36:05.678 INFO:teuthology.packaging:ref: None 2024-03-24T09:36:05.678 INFO:teuthology.packaging:tag: None 2024-03-24T09:36:05.678 INFO:teuthology.packaging:branch: reef 2024-03-24T09:36:05.678 INFO:teuthology.packaging:sha1: 21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T09:36:05.678 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-03-24T09:36:05.680 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'reef', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de', 'tag': None, 'wait_for_package': False} 2024-03-24T09:36:05.680 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T09:36:05.681 INFO:teuthology.packaging:ref: None 2024-03-24T09:36:05.681 INFO:teuthology.packaging:tag: None 2024-03-24T09:36:05.681 INFO:teuthology.packaging:branch: reef 2024-03-24T09:36:05.681 INFO:teuthology.packaging:sha1: 21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T09:36:05.681 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-03-24T09:36:06.006 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/reef/738c2fd67dbc0b17e2e99568a03974712d998229/centos/9/flavors/default/ 2024-03-24T09:36:06.007 INFO:teuthology.task.install.rpm:Package version is 18.2.2-952.g738c2fd6 2024-03-24T09:36:06.026 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/reef/738c2fd67dbc0b17e2e99568a03974712d998229/centos/9/flavors/default/ 2024-03-24T09:36:06.027 INFO:teuthology.task.install.rpm:Package version is 18.2.2-952.g738c2fd6 2024-03-24T09:36:06.183 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/reef/738c2fd67dbc0b17e2e99568a03974712d998229/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/738c2fd67dbc0b17e2e99568a03974712d998229/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/738c2fd67dbc0b17e2e99568a03974712d998229/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T09:36:06.183 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T09:36:06.183 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T09:36:06.198 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/reef/738c2fd67dbc0b17e2e99568a03974712d998229/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/738c2fd67dbc0b17e2e99568a03974712d998229/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/738c2fd67dbc0b17e2e99568a03974712d998229/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T09:36:06.198 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-24T09:36:06.198 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T09:36:06.212 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-24T09:36:06.213 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T09:36:06.213 INFO:teuthology.packaging:ref: None 2024-03-24T09:36:06.213 INFO:teuthology.packaging:tag: None 2024-03-24T09:36:06.213 INFO:teuthology.packaging:branch: reef 2024-03-24T09:36:06.213 INFO:teuthology.packaging:sha1: 21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T09:36:06.213 DEBUG:teuthology.orchestra.run.smithi022:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/reef/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T09:36:06.227 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-24T09:36:06.228 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T09:36:06.228 INFO:teuthology.packaging:ref: None 2024-03-24T09:36:06.228 INFO:teuthology.packaging:tag: None 2024-03-24T09:36:06.228 INFO:teuthology.packaging:branch: reef 2024-03-24T09:36:06.228 INFO:teuthology.packaging:sha1: 21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T09:36:06.228 DEBUG:teuthology.orchestra.run.smithi155:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/reef/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T09:36:06.282 DEBUG:teuthology.orchestra.run.smithi022:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T09:36:06.296 DEBUG:teuthology.orchestra.run.smithi155:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T09:36:06.362 DEBUG:teuthology.orchestra.run.smithi022:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T09:36:06.377 DEBUG:teuthology.orchestra.run.smithi155:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T09:36:06.429 INFO:teuthology.orchestra.run.smithi022.stdout:check_obsoletes = 1 2024-03-24T09:36:06.431 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum clean all 2024-03-24T09:36:06.443 INFO:teuthology.orchestra.run.smithi155.stdout:check_obsoletes = 1 2024-03-24T09:36:06.445 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum clean all 2024-03-24T09:36:06.687 INFO:teuthology.orchestra.run.smithi155.stdout:74 files removed 2024-03-24T09:36:06.714 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-radosgw 2024-03-24T09:36:06.718 INFO:teuthology.orchestra.run.smithi022.stdout:74 files removed 2024-03-24T09:36:06.746 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-radosgw 2024-03-24T09:36:07.470 INFO:teuthology.orchestra.run.smithi022.stdout:ceph packages for x86_64 186 kB/s | 77 kB 00:00 2024-03-24T09:36:07.472 INFO:teuthology.orchestra.run.smithi155.stdout:ceph packages for x86_64 169 kB/s | 77 kB 00:00 2024-03-24T09:36:07.845 INFO:teuthology.orchestra.run.smithi022.stdout:ceph noarch packages 31 kB/s | 9.9 kB 00:00 2024-03-24T09:36:07.857 INFO:teuthology.orchestra.run.smithi155.stdout:ceph noarch packages 30 kB/s | 9.9 kB 00:00 2024-03-24T09:36:08.187 INFO:teuthology.orchestra.run.smithi022.stdout:ceph source packages 6.2 kB/s | 1.9 kB 00:00 2024-03-24T09:36:08.207 INFO:teuthology.orchestra.run.smithi155.stdout:ceph source packages 6.0 kB/s | 1.9 kB 00:00 2024-03-24T09:36:08.792 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-24T09:36:08.890 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-24T09:36:10.252 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS-9 - BaseOS 6.1 MB/s | 2.0 MB 00:00 2024-03-24T09:36:10.466 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS-9 - BaseOS 5.2 MB/s | 2.0 MB 00:00 2024-03-24T09:36:11.009 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.0 MB 00:00 2024-03-24T09:36:11.465 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-24T09:36:12.868 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-03-24T09:36:17.321 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - CRB 6.3 MB/s | 6.1 MB 00:00 2024-03-24T09:36:18.695 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - Extras packages 72 kB/s | 16 kB 00:00 2024-03-24T09:36:20.115 INFO:teuthology.orchestra.run.smithi155.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-24T09:36:25.030 INFO:teuthology.orchestra.run.smithi155.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-24T09:36:27.466 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-24T09:36:27.482 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-24T09:36:27.482 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-24T09:36:27.482 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-24T09:36:27.483 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-24T09:36:27.483 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 7.7 M 2024-03-24T09:36:27.483 INFO:teuthology.orchestra.run.smithi155.stdout:Upgrading: 2024-03-24T09:36:27.483 INFO:teuthology.orchestra.run.smithi155.stdout: librados2 x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 3.3 M 2024-03-24T09:36:27.483 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1 x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 3.0 M 2024-03-24T09:36:27.483 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-24T09:36:27.483 INFO:teuthology.orchestra.run.smithi155.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-24T09:36:27.483 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 5.1 M 2024-03-24T09:36:27.483 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 18 M 2024-03-24T09:36:27.484 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-selinux x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 25 k 2024-03-24T09:36:27.484 INFO:teuthology.orchestra.run.smithi155.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-03-24T09:36:27.484 INFO:teuthology.orchestra.run.smithi155.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T09:36:27.484 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-24T09:36:27.484 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-24T09:36:27.484 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2 x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 702 k 2024-03-24T09:36:27.484 INFO:teuthology.orchestra.run.smithi155.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T09:36:27.484 INFO:teuthology.orchestra.run.smithi155.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T09:36:27.484 INFO:teuthology.orchestra.run.smithi155.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T09:36:27.485 INFO:teuthology.orchestra.run.smithi155.stdout: libradosstriper1 x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 474 k 2024-03-24T09:36:27.485 INFO:teuthology.orchestra.run.smithi155.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T09:36:27.485 INFO:teuthology.orchestra.run.smithi155.stdout: librgw2 x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 4.5 M 2024-03-24T09:36:27.485 INFO:teuthology.orchestra.run.smithi155.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T09:36:27.485 INFO:teuthology.orchestra.run.smithi155.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T09:36:27.485 INFO:teuthology.orchestra.run.smithi155.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-24T09:36:27.485 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-argparse x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 45 k 2024-03-24T09:36:27.485 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-common x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 127 k 2024-03-24T09:36:27.485 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 161 k 2024-03-24T09:36:27.485 INFO:teuthology.orchestra.run.smithi155.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T09:36:27.486 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 321 k 2024-03-24T09:36:27.486 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 298 k 2024-03-24T09:36:27.486 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 99 k 2024-03-24T09:36:27.486 INFO:teuthology.orchestra.run.smithi155.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-24T09:36:27.486 INFO:teuthology.orchestra.run.smithi155.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T09:36:27.486 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:36:27.486 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-24T09:36:27.486 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-24T09:36:27.486 INFO:teuthology.orchestra.run.smithi155.stdout:Install 28 Packages 2024-03-24T09:36:27.487 INFO:teuthology.orchestra.run.smithi155.stdout:Upgrade 2 Packages 2024-03-24T09:36:27.487 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:36:27.487 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 53 M 2024-03-24T09:36:27.487 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-24T09:36:29.255 INFO:teuthology.orchestra.run.smithi155.stdout:(1/30): ceph-common-18.2.2-952.g738c2fd6.el9.x8 11 MB/s | 18 MB 00:01 2024-03-24T09:36:29.331 INFO:teuthology.orchestra.run.smithi155.stdout:(2/30): ceph-base-18.2.2-952.g738c2fd6.el9.x86_ 2.8 MB/s | 5.1 MB 00:01 2024-03-24T09:36:29.347 INFO:teuthology.orchestra.run.smithi155.stdout:(3/30): ceph-selinux-18.2.2-952.g738c2fd6.el9.x 274 kB/s | 25 kB 00:00 2024-03-24T09:36:29.406 INFO:teuthology.orchestra.run.smithi155.stdout:(4/30): libradosstriper1-18.2.2-952.g738c2fd6.e 7.9 MB/s | 474 kB 00:00 2024-03-24T09:36:29.473 INFO:teuthology.orchestra.run.smithi155.stdout:(5/30): libcephfs2-18.2.2-952.g738c2fd6.el9.x86 4.8 MB/s | 702 kB 00:00 2024-03-24T09:36:29.658 INFO:teuthology.orchestra.run.smithi155.stdout:(6/30): ceph-radosgw-18.2.2-952.g738c2fd6.el9.x 3.6 MB/s | 7.7 MB 00:02 2024-03-24T09:36:29.673 INFO:teuthology.orchestra.run.smithi155.stdout:(7/30): python3-ceph-argparse-18.2.2-952.g738c2 224 kB/s | 45 kB 00:00 2024-03-24T09:36:29.741 INFO:teuthology.orchestra.run.smithi155.stdout:(8/30): librgw2-18.2.2-952.g738c2fd6.el9.x86_64 13 MB/s | 4.5 MB 00:00 2024-03-24T09:36:29.766 INFO:teuthology.orchestra.run.smithi155.stdout:(9/30): python3-ceph-common-18.2.2-952.g738c2fd 1.2 MB/s | 127 kB 00:00 2024-03-24T09:36:29.791 INFO:teuthology.orchestra.run.smithi155.stdout:(10/30): python3-cephfs-18.2.2-952.g738c2fd6.el 1.3 MB/s | 161 kB 00:00 2024-03-24T09:36:29.816 INFO:teuthology.orchestra.run.smithi155.stdout:(11/30): python3-rados-18.2.2-952.g738c2fd6.el9 4.2 MB/s | 321 kB 00:00 2024-03-24T09:36:29.841 INFO:teuthology.orchestra.run.smithi155.stdout:(12/30): python3-rbd-18.2.2-952.g738c2fd6.el9.x 3.9 MB/s | 298 kB 00:00 2024-03-24T09:36:29.867 INFO:teuthology.orchestra.run.smithi155.stdout:(13/30): python3-rgw-18.2.2-952.g738c2fd6.el9.x 1.3 MB/s | 99 kB 00:00 2024-03-24T09:36:29.992 INFO:teuthology.orchestra.run.smithi155.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 603 kB/s | 106 kB 00:00 2024-03-24T09:36:30.017 INFO:teuthology.orchestra.run.smithi155.stdout:(15/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 905 kB/s | 159 kB 00:00 2024-03-24T09:36:30.043 INFO:teuthology.orchestra.run.smithi155.stdout:(16/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 348 kB/s | 44 kB 00:00 2024-03-24T09:36:30.143 INFO:teuthology.orchestra.run.smithi155.stdout:(17/30): librdkafka-1.6.1-102.el9.x86_64.rpm 4.3 MB/s | 661 kB 00:00 2024-03-24T09:36:30.168 INFO:teuthology.orchestra.run.smithi155.stdout:(18/30): python3-prettytable-0.7.2-27.el9.noarc 330 kB/s | 41 kB 00:00 2024-03-24T09:36:30.193 INFO:teuthology.orchestra.run.smithi155.stdout:(19/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.8 MB/s | 282 kB 00:00 2024-03-24T09:36:30.494 INFO:teuthology.orchestra.run.smithi155.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 316 kB/s | 111 kB 00:00 2024-03-24T09:36:30.577 INFO:teuthology.orchestra.run.smithi155.stdout:(21/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 313 kB/s | 26 kB 00:00 2024-03-24T09:36:30.602 INFO:teuthology.orchestra.run.smithi155.stdout:(22/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 712 kB/s | 309 kB 00:00 2024-03-24T09:36:30.669 INFO:teuthology.orchestra.run.smithi155.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 528 kB/s | 49 kB 00:00 2024-03-24T09:36:30.695 INFO:teuthology.orchestra.run.smithi155.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 732 kB/s | 67 kB 00:00 2024-03-24T09:36:30.804 INFO:teuthology.orchestra.run.smithi155.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-03-24T09:36:30.870 INFO:teuthology.orchestra.run.smithi155.stdout:(26/30): libarrow-9.0.0-10.el9.x86_64.rpm 6.5 MB/s | 4.4 MB 00:00 2024-03-24T09:36:30.920 INFO:teuthology.orchestra.run.smithi155.stdout:(27/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.3 MB/s | 838 kB 00:00 2024-03-24T09:36:31.021 INFO:teuthology.orchestra.run.smithi155.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 7.3 MB/s | 1.6 MB 00:00 2024-03-24T09:36:31.205 INFO:teuthology.orchestra.run.smithi155.stdout:(29/30): librbd1-18.2.2-952.g738c2fd6.el9.x86_6 11 MB/s | 3.0 MB 00:00 2024-03-24T09:36:31.488 INFO:teuthology.orchestra.run.smithi155.stdout:(30/30): librados2-18.2.2-952.g738c2fd6.el9.x86 5.3 MB/s | 3.3 MB 00:00 2024-03-24T09:36:31.495 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:36:31.495 INFO:teuthology.orchestra.run.smithi155.stdout:Total 13 MB/s | 53 MB 00:04 2024-03-24T09:36:31.496 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-24T09:36:31.530 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-24T09:36:31.530 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-24T09:36:31.990 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-24T09:36:31.990 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-24T09:36:32.580 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - AppStream 963 kB/s | 19 MB 00:20 2024-03-24T09:36:33.379 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-24T09:36:33.463 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-03-24T09:36:33.492 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-03-24T09:36:33.559 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-03-24T09:36:33.927 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-03-24T09:36:33.966 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : librados2-2:18.2.2-952.g738c2fd6.el9.x86_64 5/32 2024-03-24T09:36:34.058 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados2-2:18.2.2-952.g738c2fd6.el9.x86_64 5/32 2024-03-24T09:36:34.141 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rados-2:18.2.2-952.g738c2fd6.el9.x86_64 6/32 2024-03-24T09:36:34.176 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephfs2-2:18.2.2-952.g738c2fd6.el9.x86_64 7/32 2024-03-24T09:36:34.222 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephfs2-2:18.2.2-952.g738c2fd6.el9.x86_64 7/32 2024-03-24T09:36:34.251 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-03-24T09:36:34.294 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ceph-argparse-2:18.2.2-952.g738c2fd6.el9.x 9/32 2024-03-24T09:36:34.361 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cephfs-2:18.2.2-952.g738c2fd6.el9.x86_64 10/32 2024-03-24T09:36:34.387 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libradosstriper1-2:18.2.2-952.g738c2fd6.el9.x86_64 11/32 2024-03-24T09:36:34.453 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libradosstriper1-2:18.2.2-952.g738c2fd6.el9.x86_64 11/32 2024-03-24T09:36:34.559 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-03-24T09:36:34.595 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-03-24T09:36:34.635 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-03-24T09:36:34.667 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-03-24T09:36:34.722 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-03-24T09:36:34.755 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-03-24T09:36:34.787 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-03-24T09:36:35.100 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-03-24T09:36:35.126 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : librbd1-2:18.2.2-952.g738c2fd6.el9.x86_64 20/32 2024-03-24T09:36:35.199 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librbd1-2:18.2.2-952.g738c2fd6.el9.x86_64 20/32 2024-03-24T09:36:35.230 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rbd-2:18.2.2-952.g738c2fd6.el9.x86_64 21/32 2024-03-24T09:36:35.275 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-03-24T09:36:35.666 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-03-24T09:36:35.695 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librgw2-2:18.2.2-952.g738c2fd6.el9.x86_64 24/32 2024-03-24T09:36:35.767 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librgw2-2:18.2.2-952.g738c2fd6.el9.x86_64 24/32 2024-03-24T09:36:35.843 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rgw-2:18.2.2-952.g738c2fd6.el9.x86_64 25/32 2024-03-24T09:36:35.872 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ceph-common-2:18.2.2-952.g738c2fd6.el9.x86 26/32 2024-03-24T09:36:37.944 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-common-2:18.2.2-952.g738c2fd6.el9.x86_64 27/32 2024-03-24T09:36:37.985 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-common-2:18.2.2-952.g738c2fd6.el9.x86_64 27/32 2024-03-24T09:36:38.055 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - CRB 3.4 MB/s | 6.1 MB 00:01 2024-03-24T09:36:38.519 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-common-2:18.2.2-952.g738c2fd6.el9.x86_64 27/32 2024-03-24T09:36:38.555 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-base-2:18.2.2-952.g738c2fd6.el9.x86_64 28/32 2024-03-24T09:36:38.648 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-base-2:18.2.2-952.g738c2fd6.el9.x86_64 28/32 2024-03-24T09:36:38.649 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T09:36:38.649 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T09:36:38.649 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:36:38.690 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-selinux-2:18.2.2-952.g738c2fd6.el9.x86_64 29/32 2024-03-24T09:36:39.456 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - Extras packages 87 kB/s | 16 kB 00:00 2024-03-24T09:36:40.282 INFO:teuthology.orchestra.run.smithi022.stdout:Extra Packages for Enterprise Linux 29 MB/s | 21 MB 00:00 2024-03-24T09:36:45.383 INFO:teuthology.orchestra.run.smithi022.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-24T09:36:47.929 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-24T09:36:47.939 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T09:36:47.940 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-24T09:36:47.940 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T09:36:47.940 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-24T09:36:47.940 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-radosgw x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 7.7 M 2024-03-24T09:36:47.940 INFO:teuthology.orchestra.run.smithi022.stdout:Upgrading: 2024-03-24T09:36:47.940 INFO:teuthology.orchestra.run.smithi022.stdout: librados2 x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 3.3 M 2024-03-24T09:36:47.940 INFO:teuthology.orchestra.run.smithi022.stdout: librbd1 x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 3.0 M 2024-03-24T09:36:47.940 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-24T09:36:47.940 INFO:teuthology.orchestra.run.smithi022.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-24T09:36:47.940 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-base x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 5.1 M 2024-03-24T09:36:47.941 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-common x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 18 M 2024-03-24T09:36:47.941 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-selinux x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 25 k 2024-03-24T09:36:47.941 INFO:teuthology.orchestra.run.smithi022.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-03-24T09:36:47.941 INFO:teuthology.orchestra.run.smithi022.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T09:36:47.941 INFO:teuthology.orchestra.run.smithi022.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-24T09:36:47.941 INFO:teuthology.orchestra.run.smithi022.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-24T09:36:47.941 INFO:teuthology.orchestra.run.smithi022.stdout: libcephfs2 x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 702 k 2024-03-24T09:36:47.941 INFO:teuthology.orchestra.run.smithi022.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T09:36:47.941 INFO:teuthology.orchestra.run.smithi022.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T09:36:47.941 INFO:teuthology.orchestra.run.smithi022.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T09:36:47.941 INFO:teuthology.orchestra.run.smithi022.stdout: libradosstriper1 x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 474 k 2024-03-24T09:36:47.942 INFO:teuthology.orchestra.run.smithi022.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T09:36:47.942 INFO:teuthology.orchestra.run.smithi022.stdout: librgw2 x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 4.5 M 2024-03-24T09:36:47.942 INFO:teuthology.orchestra.run.smithi022.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T09:36:47.942 INFO:teuthology.orchestra.run.smithi022.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T09:36:47.942 INFO:teuthology.orchestra.run.smithi022.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-24T09:36:47.942 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ceph-argparse x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 45 k 2024-03-24T09:36:47.942 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ceph-common x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 127 k 2024-03-24T09:36:47.942 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cephfs x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 161 k 2024-03-24T09:36:47.942 INFO:teuthology.orchestra.run.smithi022.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T09:36:47.942 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rados x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 321 k 2024-03-24T09:36:47.942 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rbd x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 298 k 2024-03-24T09:36:47.943 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rgw x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 99 k 2024-03-24T09:36:47.943 INFO:teuthology.orchestra.run.smithi022.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-24T09:36:47.943 INFO:teuthology.orchestra.run.smithi022.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T09:36:47.943 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:36:47.943 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-24T09:36:47.943 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T09:36:47.943 INFO:teuthology.orchestra.run.smithi022.stdout:Install 28 Packages 2024-03-24T09:36:47.943 INFO:teuthology.orchestra.run.smithi022.stdout:Upgrade 2 Packages 2024-03-24T09:36:47.943 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:36:47.944 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 53 M 2024-03-24T09:36:47.944 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-24T09:36:48.366 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-selinux-2:18.2.2-952.g738c2fd6.el9.x86_64 29/32 2024-03-24T09:36:48.367 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /sys 2024-03-24T09:36:48.367 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /proc 2024-03-24T09:36:48.367 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /mnt 2024-03-24T09:36:48.367 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /var/tmp 2024-03-24T09:36:48.367 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /home 2024-03-24T09:36:48.367 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /root 2024-03-24T09:36:48.367 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /tmp 2024-03-24T09:36:48.367 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:36:48.809 INFO:teuthology.orchestra.run.smithi022.stdout:(1/30): ceph-base-18.2.2-952.g738c2fd6.el9.x86_ 7.0 MB/s | 5.1 MB 00:00 2024-03-24T09:36:48.867 INFO:teuthology.orchestra.run.smithi022.stdout:(2/30): ceph-selinux-18.2.2-952.g738c2fd6.el9.x 431 kB/s | 25 kB 00:00 2024-03-24T09:36:49.034 INFO:teuthology.orchestra.run.smithi022.stdout:(3/30): ceph-radosgw-18.2.2-952.g738c2fd6.el9.x 8.1 MB/s | 7.7 MB 00:00 2024-03-24T09:36:49.059 INFO:teuthology.orchestra.run.smithi022.stdout:(4/30): libcephfs2-18.2.2-952.g738c2fd6.el9.x86 3.6 MB/s | 702 kB 00:00 2024-03-24T09:36:49.101 INFO:teuthology.orchestra.run.smithi022.stdout:(5/30): libradosstriper1-18.2.2-952.g738c2fd6.e 6.9 MB/s | 474 kB 00:00 2024-03-24T09:36:49.117 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-radosgw-2:18.2.2-952.g738c2fd6.el9.x86_64 30/32 2024-03-24T09:36:49.143 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-952.g738c2fd6.el9.x86_64 30/32 2024-03-24T09:36:49.143 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:36:49.143 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T09:36:49.143 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T09:36:49.143 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T09:36:49.144 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:36:49.145 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-03-24T09:36:49.151 INFO:teuthology.orchestra.run.smithi022.stdout:(6/30): python3-ceph-argparse-18.2.2-952.g738c2 902 kB/s | 45 kB 00:00 2024-03-24T09:36:49.210 INFO:teuthology.orchestra.run.smithi022.stdout:(7/30): python3-ceph-common-18.2.2-952.g738c2fd 2.1 MB/s | 127 kB 00:00 2024-03-24T09:36:49.220 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-03-24T09:36:49.220 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-03-24T09:36:49.277 INFO:teuthology.orchestra.run.smithi022.stdout:(8/30): python3-cephfs-18.2.2-952.g738c2fd6.el9 2.4 MB/s | 161 kB 00:00 2024-03-24T09:36:49.352 INFO:teuthology.orchestra.run.smithi022.stdout:(9/30): python3-rados-18.2.2-952.g738c2fd6.el9. 4.2 MB/s | 321 kB 00:00 2024-03-24T09:36:49.436 INFO:teuthology.orchestra.run.smithi022.stdout:(10/30): librgw2-18.2.2-952.g738c2fd6.el9.x86_6 12 MB/s | 4.5 MB 00:00 2024-03-24T09:36:49.461 INFO:teuthology.orchestra.run.smithi022.stdout:(11/30): python3-rbd-18.2.2-952.g738c2fd6.el9.x 2.7 MB/s | 298 kB 00:00 2024-03-24T09:36:49.495 INFO:teuthology.orchestra.run.smithi022.stdout:(12/30): python3-rgw-18.2.2-952.g738c2fd6.el9.x 1.7 MB/s | 99 kB 00:00 2024-03-24T09:36:49.612 INFO:teuthology.orchestra.run.smithi022.stdout:(13/30): boost-program-options-1.75.0-8.el9.x86 704 kB/s | 106 kB 00:00 2024-03-24T09:36:49.637 INFO:teuthology.orchestra.run.smithi022.stdout:(14/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 159 kB 00:00 2024-03-24T09:36:49.654 INFO:teuthology.orchestra.run.smithi022.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-24T09:36:49.704 INFO:teuthology.orchestra.run.smithi022.stdout:(16/30): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-24T09:36:49.729 INFO:teuthology.orchestra.run.smithi022.stdout:(17/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-24T09:36:49.746 INFO:teuthology.orchestra.run.smithi022.stdout:(18/30): python3-prettytable-0.7.2-27.el9.noarc 999 kB/s | 41 kB 00:00 2024-03-24T09:36:49.863 INFO:teuthology.orchestra.run.smithi022.stdout:(19/30): fmt-8.1.1-5.el9.x86_64.rpm 829 kB/s | 111 kB 00:00 2024-03-24T09:36:49.914 INFO:teuthology.orchestra.run.smithi022.stdout:(20/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.8 MB/s | 309 kB 00:00 2024-03-24T09:36:50.098 INFO:teuthology.orchestra.run.smithi022.stdout:(21/30): ceph-common-18.2.2-952.g738c2fd6.el9.x 9.1 MB/s | 18 MB 00:02 2024-03-24T09:36:50.123 INFO:teuthology.orchestra.run.smithi022.stdout:(22/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 124 kB/s | 26 kB 00:00 2024-03-24T09:36:50.165 INFO:teuthology.orchestra.run.smithi022.stdout:(23/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-03-24T09:36:50.198 INFO:teuthology.orchestra.run.smithi022.stdout:(24/30): liboath-2.6.7-2.el9.x86_64.rpm 483 kB/s | 49 kB 00:00 2024-03-24T09:36:50.293 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-03-24T09:36:50.293 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-base-2:18.2.2-952.g738c2fd6.el9.x86_64 1/32 2024-03-24T09:36:50.293 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-common-2:18.2.2-952.g738c2fd6.el9.x86_64 2/32 2024-03-24T09:36:50.293 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-radosgw-2:18.2.2-952.g738c2fd6.el9.x86_64 3/32 2024-03-24T09:36:50.293 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-selinux-2:18.2.2-952.g738c2fd6.el9.x86_64 4/32 2024-03-24T09:36:50.294 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephfs2-2:18.2.2-952.g738c2fd6.el9.x86_64 5/32 2024-03-24T09:36:50.294 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libradosstriper1-2:18.2.2-952.g738c2fd6.el9.x86_64 6/32 2024-03-24T09:36:50.294 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librgw2-2:18.2.2-952.g738c2fd6.el9.x86_64 7/32 2024-03-24T09:36:50.294 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ceph-argparse-2:18.2.2-952.g738c2fd6.el9.x 8/32 2024-03-24T09:36:50.294 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ceph-common-2:18.2.2-952.g738c2fd6.el9.x86 9/32 2024-03-24T09:36:50.294 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cephfs-2:18.2.2-952.g738c2fd6.el9.x86_64 10/32 2024-03-24T09:36:50.294 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rados-2:18.2.2-952.g738c2fd6.el9.x86_64 11/32 2024-03-24T09:36:50.294 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rbd-2:18.2.2-952.g738c2fd6.el9.x86_64 12/32 2024-03-24T09:36:50.294 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rgw-2:18.2.2-952.g738c2fd6.el9.x86_64 13/32 2024-03-24T09:36:50.294 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-03-24T09:36:50.294 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-03-24T09:36:50.294 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-03-24T09:36:50.294 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-03-24T09:36:50.295 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-03-24T09:36:50.295 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-03-24T09:36:50.295 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-03-24T09:36:50.295 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-03-24T09:36:50.295 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-03-24T09:36:50.295 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-03-24T09:36:50.295 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-03-24T09:36:50.295 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-03-24T09:36:50.295 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-03-24T09:36:50.295 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-03-24T09:36:50.295 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-03-24T09:36:50.295 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados2-2:18.2.2-952.g738c2fd6.el9.x86_64 29/32 2024-03-24T09:36:50.295 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-03-24T09:36:50.295 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librbd1-2:18.2.2-952.g738c2fd6.el9.x86_64 31/32 2024-03-24T09:36:50.349 INFO:teuthology.orchestra.run.smithi022.stdout:(25/30): libarrow-9.0.0-10.el9.x86_64.rpm 9.1 MB/s | 4.4 MB 00:00 2024-03-24T09:36:50.374 INFO:teuthology.orchestra.run.smithi022.stdout:(26/30): re2-20211101-3.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-03-24T09:36:50.408 INFO:teuthology.orchestra.run.smithi022.stdout:(27/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.4 MB/s | 838 kB 00:00 2024-03-24T09:36:50.450 INFO:teuthology.orchestra.run.smithi022.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-03-24T09:36:50.700 INFO:teuthology.orchestra.run.smithi022.stdout:(29/30): librados2-18.2.2-952.g738c2fd6.el9.x86 10 MB/s | 3.3 MB 00:00 2024-03-24T09:36:50.750 INFO:teuthology.orchestra.run.smithi022.stdout:(30/30): librbd1-18.2.2-952.g738c2fd6.el9.x86_6 8.8 MB/s | 3.0 MB 00:00 2024-03-24T09:36:50.754 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:36:50.754 INFO:teuthology.orchestra.run.smithi022.stdout:Total 19 MB/s | 53 MB 00:02 2024-03-24T09:36:50.754 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-24T09:36:50.793 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-24T09:36:50.793 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-24T09:36:51.192 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-03-24T09:36:51.192 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:36:51.192 INFO:teuthology.orchestra.run.smithi155.stdout:Upgraded: 2024-03-24T09:36:51.192 INFO:teuthology.orchestra.run.smithi155.stdout: librados2-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:36:51.192 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:36:51.192 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-24T09:36:51.193 INFO:teuthology.orchestra.run.smithi155.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-24T09:36:51.193 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:36:51.193 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:36:51.193 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:36:51.193 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-selinux-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:36:51.193 INFO:teuthology.orchestra.run.smithi155.stdout: fmt-8.1.1-5.el9.x86_64 2024-03-24T09:36:51.193 INFO:teuthology.orchestra.run.smithi155.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T09:36:51.193 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-24T09:36:51.193 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-24T09:36:51.193 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:36:51.193 INFO:teuthology.orchestra.run.smithi155.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T09:36:51.193 INFO:teuthology.orchestra.run.smithi155.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T09:36:51.193 INFO:teuthology.orchestra.run.smithi155.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T09:36:51.193 INFO:teuthology.orchestra.run.smithi155.stdout: libradosstriper1-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:36:51.193 INFO:teuthology.orchestra.run.smithi155.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T09:36:51.194 INFO:teuthology.orchestra.run.smithi155.stdout: librgw2-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:36:51.194 INFO:teuthology.orchestra.run.smithi155.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T09:36:51.194 INFO:teuthology.orchestra.run.smithi155.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T09:36:51.194 INFO:teuthology.orchestra.run.smithi155.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-24T09:36:51.194 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-argparse-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:36:51.194 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-common-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:36:51.194 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:36:51.194 INFO:teuthology.orchestra.run.smithi155.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T09:36:51.194 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:36:51.194 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:36:51.194 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:36:51.195 INFO:teuthology.orchestra.run.smithi155.stdout: re2-1:20211101-3.el9.x86_64 2024-03-24T09:36:51.195 INFO:teuthology.orchestra.run.smithi155.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T09:36:51.195 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:36:51.195 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-24T09:36:51.264 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-24T09:36:51.264 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-24T09:36:51.553 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-test 2024-03-24T09:36:52.116 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:27 ago on Sun 24 Mar 2024 09:36:25 AM UTC. 2024-03-24T09:36:52.264 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-24T09:36:52.265 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-24T09:36:52.265 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-24T09:36:52.265 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-24T09:36:52.265 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-24T09:36:52.265 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 40 M 2024-03-24T09:36:52.265 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-24T09:36:52.265 INFO:teuthology.orchestra.run.smithi155.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T09:36:52.265 INFO:teuthology.orchestra.run.smithi155.stdout: libcephsqlite x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 166 k 2024-03-24T09:36:52.265 INFO:teuthology.orchestra.run.smithi155.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T09:36:52.265 INFO:teuthology.orchestra.run.smithi155.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T09:36:52.266 INFO:teuthology.orchestra.run.smithi155.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T09:36:52.266 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:36:52.266 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-24T09:36:52.266 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-24T09:36:52.266 INFO:teuthology.orchestra.run.smithi155.stdout:Install 6 Packages 2024-03-24T09:36:52.266 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:36:52.266 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 41 M 2024-03-24T09:36:52.267 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 165 M 2024-03-24T09:36:52.267 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-24T09:36:52.473 INFO:teuthology.orchestra.run.smithi155.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-24T09:36:52.540 INFO:teuthology.orchestra.run.smithi155.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 4.4 MB/s | 300 kB 00:00 2024-03-24T09:36:52.582 INFO:teuthology.orchestra.run.smithi155.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-24T09:36:52.607 INFO:teuthology.orchestra.run.smithi155.stdout:(4/6): libcephsqlite-18.2.2-952.g738c2fd6.el9.x 558 kB/s | 166 kB 00:00 2024-03-24T09:36:52.757 INFO:teuthology.orchestra.run.smithi155.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-03-24T09:36:52.793 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-24T09:36:52.885 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-03-24T09:36:52.921 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-03-24T09:36:52.987 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-03-24T09:36:53.376 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-03-24T09:36:53.412 INFO:teuthology.orchestra.run.smithi022.stdout: Upgrading : librados2-2:18.2.2-952.g738c2fd6.el9.x86_64 5/32 2024-03-24T09:36:53.505 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librados2-2:18.2.2-952.g738c2fd6.el9.x86_64 5/32 2024-03-24T09:36:53.590 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-rados-2:18.2.2-952.g738c2fd6.el9.x86_64 6/32 2024-03-24T09:36:53.630 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libcephfs2-2:18.2.2-952.g738c2fd6.el9.x86_64 7/32 2024-03-24T09:36:53.693 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: libcephfs2-2:18.2.2-952.g738c2fd6.el9.x86_64 7/32 2024-03-24T09:36:53.722 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-03-24T09:36:53.766 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-ceph-argparse-2:18.2.2-952.g738c2fd6.el9.x 9/32 2024-03-24T09:36:53.833 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cephfs-2:18.2.2-952.g738c2fd6.el9.x86_64 10/32 2024-03-24T09:36:53.857 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libradosstriper1-2:18.2.2-952.g738c2fd6.el9.x86_64 11/32 2024-03-24T09:36:53.915 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: libradosstriper1-2:18.2.2-952.g738c2fd6.el9.x86_64 11/32 2024-03-24T09:36:54.024 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-03-24T09:36:54.057 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-03-24T09:36:54.097 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-03-24T09:36:54.138 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-03-24T09:36:54.165 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-03-24T09:36:54.209 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-03-24T09:36:54.242 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-03-24T09:36:54.570 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-03-24T09:36:54.574 INFO:teuthology.orchestra.run.smithi155.stdout:(6/6): ceph-test-18.2.2-952.g738c2fd6.el9.x86_6 18 MB/s | 40 MB 00:02 2024-03-24T09:36:54.577 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:36:54.578 INFO:teuthology.orchestra.run.smithi155.stdout:Total 18 MB/s | 41 MB 00:02 2024-03-24T09:36:54.592 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-24T09:36:54.613 INFO:teuthology.orchestra.run.smithi022.stdout: Upgrading : librbd1-2:18.2.2-952.g738c2fd6.el9.x86_64 20/32 2024-03-24T09:36:54.613 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-24T09:36:54.613 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-24T09:36:54.703 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librbd1-2:18.2.2-952.g738c2fd6.el9.x86_64 20/32 2024-03-24T09:36:54.751 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-rbd-2:18.2.2-952.g738c2fd6.el9.x86_64 21/32 2024-03-24T09:36:54.794 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-03-24T09:36:54.948 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-24T09:36:54.948 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-24T09:36:55.217 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-03-24T09:36:55.249 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : librgw2-2:18.2.2-952.g738c2fd6.el9.x86_64 24/32 2024-03-24T09:36:55.320 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librgw2-2:18.2.2-952.g738c2fd6.el9.x86_64 24/32 2024-03-24T09:36:55.398 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-rgw-2:18.2.2-952.g738c2fd6.el9.x86_64 25/32 2024-03-24T09:36:55.426 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-ceph-common-2:18.2.2-952.g738c2fd6.el9.x86 26/32 2024-03-24T09:36:55.493 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-24T09:36:55.575 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-24T09:36:55.627 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-24T09:36:55.675 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-24T09:36:55.716 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-24T09:36:55.744 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephsqlite-2:18.2.2-952.g738c2fd6.el9.x86_64 5/6 2024-03-24T09:36:57.542 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-common-2:18.2.2-952.g738c2fd6.el9.x86_64 27/32 2024-03-24T09:36:57.581 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-common-2:18.2.2-952.g738c2fd6.el9.x86_64 27/32 2024-03-24T09:36:58.140 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-common-2:18.2.2-952.g738c2fd6.el9.x86_64 27/32 2024-03-24T09:36:58.184 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-base-2:18.2.2-952.g738c2fd6.el9.x86_64 28/32 2024-03-24T09:36:58.274 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-base-2:18.2.2-952.g738c2fd6.el9.x86_64 28/32 2024-03-24T09:36:58.274 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T09:36:58.275 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T09:36:58.275 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:36:58.310 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-selinux-2:18.2.2-952.g738c2fd6.el9.x86_64 29/32 2024-03-24T09:36:59.479 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephsqlite-2:18.2.2-952.g738c2fd6.el9.x86_64 5/6 2024-03-24T09:36:59.551 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-test-2:18.2.2-952.g738c2fd6.el9.x86_64 6/6 2024-03-24T09:37:01.619 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-test-2:18.2.2-952.g738c2fd6.el9.x86_64 6/6 2024-03-24T09:37:01.619 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-test-2:18.2.2-952.g738c2fd6.el9.x86_64 1/6 2024-03-24T09:37:01.619 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephsqlite-2:18.2.2-952.g738c2fd6.el9.x86_64 2/6 2024-03-24T09:37:01.619 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-24T09:37:01.619 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-24T09:37:01.619 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-24T09:37:01.996 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-24T09:37:01.996 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:01.996 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-24T09:37:01.996 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:37:01.996 INFO:teuthology.orchestra.run.smithi155.stdout: jq-1.6-16.el9.x86_64 2024-03-24T09:37:01.996 INFO:teuthology.orchestra.run.smithi155.stdout: libcephsqlite-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:37:01.996 INFO:teuthology.orchestra.run.smithi155.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T09:37:01.996 INFO:teuthology.orchestra.run.smithi155.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T09:37:01.997 INFO:teuthology.orchestra.run.smithi155.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T09:37:01.997 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:01.997 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-24T09:37:02.282 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph 2024-03-24T09:37:02.852 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:37 ago on Sun 24 Mar 2024 09:36:25 AM UTC. 2024-03-24T09:37:03.008 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-24T09:37:03.011 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================== 2024-03-24T09:37:03.011 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-24T09:37:03.011 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================== 2024-03-24T09:37:03.011 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-24T09:37:03.011 INFO:teuthology.orchestra.run.smithi155.stdout: ceph x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 6.5 k 2024-03-24T09:37:03.011 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-24T09:37:03.011 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 2.1 M 2024-03-24T09:37:03.012 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 1.4 M 2024-03-24T09:37:03.012 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-modules-core noarch 2:18.2.2-952.g738c2fd6.el9 ceph-noarch 245 k 2024-03-24T09:37:03.012 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 4.7 M 2024-03-24T09:37:03.012 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 16 M 2024-03-24T09:37:03.012 INFO:teuthology.orchestra.run.smithi155.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T09:37:03.012 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T09:37:03.012 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T09:37:03.012 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T09:37:03.012 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T09:37:03.012 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T09:37:03.013 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T09:37:03.013 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T09:37:03.013 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T09:37:03.013 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T09:37:03.013 INFO:teuthology.orchestra.run.smithi155.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T09:37:03.013 INFO:teuthology.orchestra.run.smithi155.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T09:37:03.013 INFO:teuthology.orchestra.run.smithi155.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T09:37:03.013 INFO:teuthology.orchestra.run.smithi155.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T09:37:03.013 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T09:37:03.013 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T09:37:03.013 INFO:teuthology.orchestra.run.smithi155.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T09:37:03.013 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T09:37:03.014 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T09:37:03.014 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T09:37:03.014 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T09:37:03.014 INFO:teuthology.orchestra.run.smithi155.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T09:37:03.014 INFO:teuthology.orchestra.run.smithi155.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T09:37:03.014 INFO:teuthology.orchestra.run.smithi155.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T09:37:03.014 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T09:37:03.014 INFO:teuthology.orchestra.run.smithi155.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T09:37:03.014 INFO:teuthology.orchestra.run.smithi155.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T09:37:03.014 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:03.014 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-24T09:37:03.015 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================== 2024-03-24T09:37:03.015 INFO:teuthology.orchestra.run.smithi155.stdout:Install 33 Packages 2024-03-24T09:37:03.015 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:03.015 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 29 M 2024-03-24T09:37:03.015 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 104 M 2024-03-24T09:37:03.015 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-24T09:37:03.551 INFO:teuthology.orchestra.run.smithi155.stdout:(1/33): ceph-18.2.2-952.g738c2fd6.el9.x86_64.rp 20 kB/s | 6.5 kB 00:00 2024-03-24T09:37:03.593 INFO:teuthology.orchestra.run.smithi155.stdout:(2/33): ceph-mds-18.2.2-952.g738c2fd6.el9.x86_6 5.7 MB/s | 2.1 MB 00:00 2024-03-24T09:37:03.827 INFO:teuthology.orchestra.run.smithi155.stdout:(3/33): ceph-mgr-18.2.2-952.g738c2fd6.el9.x86_6 2.4 MB/s | 1.4 MB 00:00 2024-03-24T09:37:03.960 INFO:teuthology.orchestra.run.smithi155.stdout:(4/33): ceph-mgr-modules-core-18.2.2-952.g738c2 1.8 MB/s | 245 kB 00:00 2024-03-24T09:37:04.068 INFO:teuthology.orchestra.run.smithi155.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 144 kB 00:00 2024-03-24T09:37:04.102 INFO:teuthology.orchestra.run.smithi155.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 946 kB/s | 32 kB 00:00 2024-03-24T09:37:04.135 INFO:teuthology.orchestra.run.smithi155.stdout:(7/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-24T09:37:04.178 INFO:teuthology.orchestra.run.smithi155.stdout:(8/33): python3-toml-0.10.2-6.el9.noarch.rpm 924 kB/s | 39 kB 00:00 2024-03-24T09:37:04.228 INFO:teuthology.orchestra.run.smithi155.stdout:(9/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-24T09:37:04.405 INFO:teuthology.orchestra.run.smithi155.stdout:(10/33): ceph-osd-18.2.2-952.g738c2fd6.el9.x86_ 20 MB/s | 16 MB 00:00 2024-03-24T09:37:04.458 INFO:teuthology.orchestra.run.smithi155.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 5.0 MB/s | 1.1 MB 00:00 2024-03-24T09:37:04.481 INFO:teuthology.orchestra.run.smithi155.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 103 kB 00:00 2024-03-24T09:37:04.506 INFO:teuthology.orchestra.run.smithi155.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 2.5 MB/s | 124 kB 00:00 2024-03-24T09:37:04.531 INFO:teuthology.orchestra.run.smithi155.stdout:(14/33): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-03-24T09:37:04.732 INFO:teuthology.orchestra.run.smithi155.stdout:(15/33): python3-requests-2.25.1-8.el9.noarch.r 555 kB/s | 125 kB 00:00 2024-03-24T09:37:04.841 INFO:teuthology.orchestra.run.smithi155.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 141 kB/s | 43 kB 00:00 2024-03-24T09:37:05.141 INFO:teuthology.orchestra.run.smithi155.stdout:(17/33): ceph-mon-18.2.2-952.g738c2fd6.el9.x86_ 2.9 MB/s | 4.7 MB 00:01 2024-03-24T09:37:05.166 INFO:teuthology.orchestra.run.smithi155.stdout:(18/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 396 kB/s | 172 kB 00:00 2024-03-24T09:37:05.191 INFO:teuthology.orchestra.run.smithi155.stdout:(19/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.0 MB/s | 358 kB 00:00 2024-03-24T09:37:05.258 INFO:teuthology.orchestra.run.smithi155.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 192 kB/s | 18 kB 00:00 2024-03-24T09:37:05.292 INFO:teuthology.orchestra.run.smithi155.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 231 kB/s | 23 kB 00:00 2024-03-24T09:37:05.317 INFO:teuthology.orchestra.run.smithi155.stdout:(22/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 61 kB/s | 11 kB 00:00 2024-03-24T09:37:05.351 INFO:teuthology.orchestra.run.smithi155.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 211 kB/s | 19 kB 00:00 2024-03-24T09:37:05.384 INFO:teuthology.orchestra.run.smithi155.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 213 kB/s | 20 kB 00:00 2024-03-24T09:37:05.443 INFO:teuthology.orchestra.run.smithi155.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 857 kB/s | 79 kB 00:00 2024-03-24T09:37:05.477 INFO:teuthology.orchestra.run.smithi155.stdout:(26/33): python3-logutils-0.3.5-21.el9.noarch.r 291 kB/s | 46 kB 00:00 2024-03-24T09:37:05.502 INFO:teuthology.orchestra.run.smithi155.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-03-24T09:37:05.535 INFO:teuthology.orchestra.run.smithi155.stdout:(28/33): python3-portend-3.1.0-2.el9.noarch.rpm 178 kB/s | 16 kB 00:00 2024-03-24T09:37:05.602 INFO:teuthology.orchestra.run.smithi155.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 356 kB/s | 36 kB 00:00 2024-03-24T09:37:05.636 INFO:teuthology.orchestra.run.smithi155.stdout:(30/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 565 kB/s | 90 kB 00:00 2024-03-24T09:37:05.703 INFO:teuthology.orchestra.run.smithi155.stdout:(31/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-03-24T09:37:05.728 INFO:teuthology.orchestra.run.smithi155.stdout:(32/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.2 MB/s | 230 kB 00:00 2024-03-24T09:37:05.753 INFO:teuthology.orchestra.run.smithi155.stdout:(33/33): python3-zc-lockfile-2.0-10.el9.noarch. 171 kB/s | 20 kB 00:00 2024-03-24T09:37:05.757 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:37:05.757 INFO:teuthology.orchestra.run.smithi155.stdout:Total 11 MB/s | 29 MB 00:02 2024-03-24T09:37:05.771 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-24T09:37:05.812 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-24T09:37:05.812 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-24T09:37:06.159 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-24T09:37:06.159 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-24T09:37:07.395 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-24T09:37:07.456 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-03-24T09:37:07.497 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-03-24T09:37:07.547 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-03-24T09:37:07.640 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-03-24T09:37:07.726 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-03-24T09:37:07.767 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-03-24T09:37:07.839 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-03-24T09:37:07.893 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-03-24T09:37:07.931 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-03-24T09:37:07.980 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-03-24T09:37:08.035 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-03-24T09:37:08.234 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-03-24T09:37:08.294 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-03-24T09:37:08.332 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-03-24T09:37:08.366 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-03-24T09:37:08.397 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-03-24T09:37:08.474 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-03-24T09:37:08.480 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-selinux-2:18.2.2-952.g738c2fd6.el9.x86_64 29/32 2024-03-24T09:37:08.480 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /sys 2024-03-24T09:37:08.480 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /proc 2024-03-24T09:37:08.480 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /mnt 2024-03-24T09:37:08.480 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /var/tmp 2024-03-24T09:37:08.480 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /home 2024-03-24T09:37:08.480 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /root 2024-03-24T09:37:08.480 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /tmp 2024-03-24T09:37:08.480 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:37:08.507 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-24T09:37:08.540 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-03-24T09:37:08.573 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-03-24T09:37:08.608 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-03-24T09:37:08.642 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-03-24T09:37:08.733 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-03-24T09:37:08.781 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-03-24T09:37:08.855 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-03-24T09:37:09.003 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-03-24T09:37:09.136 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-24T09:37:09.251 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-radosgw-2:18.2.2-952.g738c2fd6.el9.x86_64 30/32 2024-03-24T09:37:09.275 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-952.g738c2fd6.el9.x86_64 30/32 2024-03-24T09:37:09.275 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:37:09.275 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T09:37:09.275 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T09:37:09.275 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T09:37:09.275 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:37:09.277 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-03-24T09:37:09.292 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-952.g738c2fd6.el9.n 28/33 2024-03-24T09:37:09.319 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-2:18.2.2-952.g738c2fd6.el9.x86_64 29/33 2024-03-24T09:37:09.345 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-2:18.2.2-952.g738c2fd6.el9.x86_64 29/33 2024-03-24T09:37:09.345 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:37:09.345 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T09:37:09.345 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T09:37:09.345 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T09:37:09.345 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:09.487 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-03-24T09:37:09.487 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-03-24T09:37:10.544 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-03-24T09:37:10.545 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-base-2:18.2.2-952.g738c2fd6.el9.x86_64 1/32 2024-03-24T09:37:10.545 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-common-2:18.2.2-952.g738c2fd6.el9.x86_64 2/32 2024-03-24T09:37:10.545 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-radosgw-2:18.2.2-952.g738c2fd6.el9.x86_64 3/32 2024-03-24T09:37:10.545 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-selinux-2:18.2.2-952.g738c2fd6.el9.x86_64 4/32 2024-03-24T09:37:10.545 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libcephfs2-2:18.2.2-952.g738c2fd6.el9.x86_64 5/32 2024-03-24T09:37:10.545 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libradosstriper1-2:18.2.2-952.g738c2fd6.el9.x86_64 6/32 2024-03-24T09:37:10.545 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librgw2-2:18.2.2-952.g738c2fd6.el9.x86_64 7/32 2024-03-24T09:37:10.545 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-ceph-argparse-2:18.2.2-952.g738c2fd6.el9.x 8/32 2024-03-24T09:37:10.545 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-ceph-common-2:18.2.2-952.g738c2fd6.el9.x86 9/32 2024-03-24T09:37:10.545 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cephfs-2:18.2.2-952.g738c2fd6.el9.x86_64 10/32 2024-03-24T09:37:10.545 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-rados-2:18.2.2-952.g738c2fd6.el9.x86_64 11/32 2024-03-24T09:37:10.545 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-rbd-2:18.2.2-952.g738c2fd6.el9.x86_64 12/32 2024-03-24T09:37:10.545 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-rgw-2:18.2.2-952.g738c2fd6.el9.x86_64 13/32 2024-03-24T09:37:10.545 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-03-24T09:37:10.545 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-03-24T09:37:10.546 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-03-24T09:37:10.546 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-03-24T09:37:10.546 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-03-24T09:37:10.546 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-03-24T09:37:10.546 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-03-24T09:37:10.546 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-03-24T09:37:10.546 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-03-24T09:37:10.546 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-03-24T09:37:10.546 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-03-24T09:37:10.546 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-03-24T09:37:10.546 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-03-24T09:37:10.546 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-03-24T09:37:10.547 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-03-24T09:37:10.547 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librados2-2:18.2.2-952.g738c2fd6.el9.x86_64 29/32 2024-03-24T09:37:10.547 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-03-24T09:37:10.547 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librbd1-2:18.2.2-952.g738c2fd6.el9.x86_64 31/32 2024-03-24T09:37:10.762 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-osd-2:18.2.2-952.g738c2fd6.el9.x86_64 30/33 2024-03-24T09:37:10.791 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-osd-2:18.2.2-952.g738c2fd6.el9.x86_64 30/33 2024-03-24T09:37:10.791 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:37:10.791 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T09:37:10.791 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T09:37:10.791 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T09:37:10.791 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:11.264 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mon-2:18.2.2-952.g738c2fd6.el9.x86_64 31/33 2024-03-24T09:37:11.288 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mon-2:18.2.2-952.g738c2fd6.el9.x86_64 31/33 2024-03-24T09:37:11.289 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:37:11.289 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T09:37:11.289 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T09:37:11.289 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T09:37:11.289 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:11.506 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mds-2:18.2.2-952.g738c2fd6.el9.x86_64 32/33 2024-03-24T09:37:11.533 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mds-2:18.2.2-952.g738c2fd6.el9.x86_64 32/33 2024-03-24T09:37:11.533 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:37:11.533 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T09:37:11.533 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T09:37:11.533 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T09:37:11.534 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:11.596 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-2:18.2.2-952.g738c2fd6.el9.x86_64 33/33 2024-03-24T09:37:11.732 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-03-24T09:37:11.733 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:37:11.733 INFO:teuthology.orchestra.run.smithi022.stdout:Upgraded: 2024-03-24T09:37:11.733 INFO:teuthology.orchestra.run.smithi022.stdout: librados2-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:37:11.733 INFO:teuthology.orchestra.run.smithi022.stdout: librbd1-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:37:11.733 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-24T09:37:11.733 INFO:teuthology.orchestra.run.smithi022.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-24T09:37:11.733 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-base-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:37:11.733 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-common-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:37:11.733 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-radosgw-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:37:11.733 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-selinux-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:37:11.733 INFO:teuthology.orchestra.run.smithi022.stdout: fmt-8.1.1-5.el9.x86_64 2024-03-24T09:37:11.733 INFO:teuthology.orchestra.run.smithi022.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T09:37:11.733 INFO:teuthology.orchestra.run.smithi022.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-24T09:37:11.734 INFO:teuthology.orchestra.run.smithi022.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-24T09:37:11.734 INFO:teuthology.orchestra.run.smithi022.stdout: libcephfs2-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:37:11.734 INFO:teuthology.orchestra.run.smithi022.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T09:37:11.734 INFO:teuthology.orchestra.run.smithi022.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T09:37:11.734 INFO:teuthology.orchestra.run.smithi022.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T09:37:11.734 INFO:teuthology.orchestra.run.smithi022.stdout: libradosstriper1-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:37:11.734 INFO:teuthology.orchestra.run.smithi022.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T09:37:11.734 INFO:teuthology.orchestra.run.smithi022.stdout: librgw2-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:37:11.734 INFO:teuthology.orchestra.run.smithi022.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T09:37:11.734 INFO:teuthology.orchestra.run.smithi022.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T09:37:11.734 INFO:teuthology.orchestra.run.smithi022.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-24T09:37:11.734 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ceph-argparse-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:37:11.734 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ceph-common-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:37:11.734 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cephfs-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:37:11.735 INFO:teuthology.orchestra.run.smithi022.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T09:37:11.735 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rados-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:37:11.735 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rbd-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:37:11.735 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rgw-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:37:11.735 INFO:teuthology.orchestra.run.smithi022.stdout: re2-1:20211101-3.el9.x86_64 2024-03-24T09:37:11.735 INFO:teuthology.orchestra.run.smithi022.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T09:37:11.735 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:37:11.735 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-24T09:37:12.090 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-test 2024-03-24T09:37:12.655 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:00:27 ago on Sun 24 Mar 2024 09:36:45 AM UTC. 2024-03-24T09:37:12.808 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-24T09:37:12.809 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T09:37:12.809 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-24T09:37:12.809 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T09:37:12.809 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-24T09:37:12.810 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-test x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 40 M 2024-03-24T09:37:12.810 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-24T09:37:12.810 INFO:teuthology.orchestra.run.smithi022.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T09:37:12.810 INFO:teuthology.orchestra.run.smithi022.stdout: libcephsqlite x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 166 k 2024-03-24T09:37:12.810 INFO:teuthology.orchestra.run.smithi022.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T09:37:12.810 INFO:teuthology.orchestra.run.smithi022.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T09:37:12.810 INFO:teuthology.orchestra.run.smithi022.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T09:37:12.810 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:37:12.810 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-24T09:37:12.810 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T09:37:12.810 INFO:teuthology.orchestra.run.smithi022.stdout:Install 6 Packages 2024-03-24T09:37:12.810 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:37:12.811 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 41 M 2024-03-24T09:37:12.811 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 165 M 2024-03-24T09:37:12.811 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-24T09:37:13.275 INFO:teuthology.orchestra.run.smithi022.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-03-24T09:37:13.350 INFO:teuthology.orchestra.run.smithi022.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 3.9 MB/s | 300 kB 00:00 2024-03-24T09:37:13.359 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-2:18.2.2-952.g738c2fd6.el9.x86_64 33/33 2024-03-24T09:37:13.359 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-2:18.2.2-952.g738c2fd6.el9.x86_64 1/33 2024-03-24T09:37:13.359 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mds-2:18.2.2-952.g738c2fd6.el9.x86_64 2/33 2024-03-24T09:37:13.359 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-2:18.2.2-952.g738c2fd6.el9.x86_64 3/33 2024-03-24T09:37:13.360 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mon-2:18.2.2-952.g738c2fd6.el9.x86_64 4/33 2024-03-24T09:37:13.360 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-osd-2:18.2.2-952.g738c2fd6.el9.x86_64 5/33 2024-03-24T09:37:13.360 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-952.g738c2fd6.el9.n 6/33 2024-03-24T09:37:13.360 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-03-24T09:37:13.360 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-03-24T09:37:13.360 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-03-24T09:37:13.360 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-03-24T09:37:13.360 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-03-24T09:37:13.360 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-03-24T09:37:13.360 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-03-24T09:37:13.360 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-03-24T09:37:13.360 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 15/33 2024-03-24T09:37:13.360 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/33 2024-03-24T09:37:13.361 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-03-24T09:37:13.361 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-24T09:37:13.361 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-03-24T09:37:13.361 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-03-24T09:37:13.361 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-03-24T09:37:13.361 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-03-24T09:37:13.361 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-03-24T09:37:13.361 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-03-24T09:37:13.361 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-03-24T09:37:13.362 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-03-24T09:37:13.362 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-24T09:37:13.362 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-03-24T09:37:13.362 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-03-24T09:37:13.363 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-03-24T09:37:13.363 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-03-24T09:37:13.363 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-03-24T09:37:13.400 INFO:teuthology.orchestra.run.smithi022.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-24T09:37:13.659 INFO:teuthology.orchestra.run.smithi022.stdout:(4/6): libcephsqlite-18.2.2-952.g738c2fd6.el9.x 288 kB/s | 166 kB 00:00 2024-03-24T09:37:13.959 INFO:teuthology.orchestra.run.smithi022.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 335 kB/s | 187 kB 00:00 2024-03-24T09:37:14.140 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-03-24T09:37:14.140 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:14.141 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-24T09:37:14.141 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:37:14.141 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:37:14.141 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:37:14.141 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-modules-core-2:18.2.2-952.g738c2fd6.el9.noarch 2024-03-24T09:37:14.141 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:37:14.141 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:37:14.141 INFO:teuthology.orchestra.run.smithi155.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T09:37:14.141 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T09:37:14.141 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T09:37:14.141 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T09:37:14.141 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T09:37:14.141 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T09:37:14.142 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T09:37:14.142 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T09:37:14.142 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T09:37:14.142 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T09:37:14.142 INFO:teuthology.orchestra.run.smithi155.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T09:37:14.142 INFO:teuthology.orchestra.run.smithi155.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T09:37:14.142 INFO:teuthology.orchestra.run.smithi155.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T09:37:14.142 INFO:teuthology.orchestra.run.smithi155.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T09:37:14.142 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T09:37:14.142 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T09:37:14.142 INFO:teuthology.orchestra.run.smithi155.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T09:37:14.142 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T09:37:14.143 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T09:37:14.143 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T09:37:14.143 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T09:37:14.143 INFO:teuthology.orchestra.run.smithi155.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T09:37:14.143 INFO:teuthology.orchestra.run.smithi155.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T09:37:14.143 INFO:teuthology.orchestra.run.smithi155.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T09:37:14.143 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T09:37:14.143 INFO:teuthology.orchestra.run.smithi155.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T09:37:14.143 INFO:teuthology.orchestra.run.smithi155.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T09:37:14.143 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:14.143 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-24T09:37:14.494 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-base 2024-03-24T09:37:15.060 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:50 ago on Sun 24 Mar 2024 09:36:25 AM UTC. 2024-03-24T09:37:15.101 INFO:teuthology.orchestra.run.smithi022.stdout:(6/6): ceph-test-18.2.2-952.g738c2fd6.el9.x86_6 20 MB/s | 40 MB 00:02 2024-03-24T09:37:15.104 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:37:15.105 INFO:teuthology.orchestra.run.smithi022.stdout:Total 18 MB/s | 41 MB 00:02 2024-03-24T09:37:15.119 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-24T09:37:15.142 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-24T09:37:15.142 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-24T09:37:15.156 INFO:teuthology.orchestra.run.smithi155.stdout:Package ceph-base-2:18.2.2-952.g738c2fd6.el9.x86_64 is already installed. 2024-03-24T09:37:15.207 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-24T09:37:15.208 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-03-24T09:37:15.208 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-24T09:37:15.271 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install cephadm 2024-03-24T09:37:15.496 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-24T09:37:15.496 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-24T09:37:15.837 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:50 ago on Sun 24 Mar 2024 09:36:25 AM UTC. 2024-03-24T09:37:15.984 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-24T09:37:15.985 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-24T09:37:15.985 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-24T09:37:15.985 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-24T09:37:15.985 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-24T09:37:15.985 INFO:teuthology.orchestra.run.smithi155.stdout: cephadm noarch 2:18.2.2-952.g738c2fd6.el9 ceph-noarch 224 k 2024-03-24T09:37:15.985 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:15.986 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-24T09:37:15.986 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-24T09:37:15.986 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-24T09:37:15.986 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:15.986 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 224 k 2024-03-24T09:37:15.986 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 215 k 2024-03-24T09:37:15.986 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-24T09:37:16.036 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-24T09:37:16.110 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-24T09:37:16.151 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-24T09:37:16.205 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-24T09:37:16.244 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-24T09:37:16.271 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libcephsqlite-2:18.2.2-952.g738c2fd6.el9.x86_64 5/6 2024-03-24T09:37:16.339 INFO:teuthology.orchestra.run.smithi155.stdout:cephadm-18.2.2-952.g738c2fd6.el9.noarch.rpm 635 kB/s | 224 kB 00:00 2024-03-24T09:37:16.339 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:37:16.340 INFO:teuthology.orchestra.run.smithi155.stdout:Total 632 kB/s | 224 kB 00:00 2024-03-24T09:37:16.340 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-24T09:37:16.345 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-24T09:37:16.345 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-24T09:37:16.348 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-24T09:37:16.348 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-24T09:37:16.416 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-24T09:37:16.809 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: cephadm-2:18.2.2-952.g738c2fd6.el9.noarch 1/1 2024-03-24T09:37:16.893 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : cephadm-2:18.2.2-952.g738c2fd6.el9.noarch 1/1 2024-03-24T09:37:17.252 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: cephadm-2:18.2.2-952.g738c2fd6.el9.noarch 1/1 2024-03-24T09:37:17.559 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : cephadm-2:18.2.2-952.g738c2fd6.el9.noarch 1/1 2024-03-24T09:37:17.559 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:17.559 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-24T09:37:17.559 INFO:teuthology.orchestra.run.smithi155.stdout: cephadm-2:18.2.2-952.g738c2fd6.el9.noarch 2024-03-24T09:37:17.559 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:17.559 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-24T09:37:17.758 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-immutable-object-cache 2024-03-24T09:37:18.333 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:53 ago on Sun 24 Mar 2024 09:36:25 AM UTC. 2024-03-24T09:37:18.483 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-24T09:37:18.483 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-24T09:37:18.484 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-03-24T09:37:18.484 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-24T09:37:18.484 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-24T09:37:18.484 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 143 k 2024-03-24T09:37:18.484 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:18.484 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-24T09:37:18.484 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-24T09:37:18.484 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-24T09:37:18.484 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:18.484 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 143 k 2024-03-24T09:37:18.484 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 395 k 2024-03-24T09:37:18.484 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-24T09:37:18.700 INFO:teuthology.orchestra.run.smithi155.stdout:ceph-immutable-object-cache-18.2.2-952.g738c2fd 660 kB/s | 143 kB 00:00 2024-03-24T09:37:18.701 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:37:18.701 INFO:teuthology.orchestra.run.smithi155.stdout:Total 656 kB/s | 143 kB 00:00 2024-03-24T09:37:18.702 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-24T09:37:18.709 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-24T09:37:18.709 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-24T09:37:18.761 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-24T09:37:18.761 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-24T09:37:18.898 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-24T09:37:18.968 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-952.g738c2fd6.e 1/1 2024-03-24T09:37:18.993 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-952.g738c2fd6.e 1/1 2024-03-24T09:37:18.993 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:37:18.993 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T09:37:18.993 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:19.998 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-952.g738c2fd6.e 1/1 2024-03-24T09:37:19.999 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:19.999 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-24T09:37:19.999 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:37:19.999 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:19.999 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-24T09:37:20.258 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr 2024-03-24T09:37:20.292 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: libcephsqlite-2:18.2.2-952.g738c2fd6.el9.x86_64 5/6 2024-03-24T09:37:20.372 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-test-2:18.2.2-952.g738c2fd6.el9.x86_64 6/6 2024-03-24T09:37:20.835 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:55 ago on Sun 24 Mar 2024 09:36:25 AM UTC. 2024-03-24T09:37:20.935 INFO:teuthology.orchestra.run.smithi155.stdout:Package ceph-mgr-2:18.2.2-952.g738c2fd6.el9.x86_64 is already installed. 2024-03-24T09:37:20.987 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-24T09:37:20.988 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-03-24T09:37:20.988 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-24T09:37:21.057 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-dashboard 2024-03-24T09:37:21.619 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:56 ago on Sun 24 Mar 2024 09:36:25 AM UTC. 2024-03-24T09:37:21.768 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-24T09:37:21.769 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-24T09:37:21.769 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-24T09:37:21.769 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-24T09:37:21.769 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-24T09:37:21.769 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard noarch 2:18.2.2-952.g738c2fd6.el9 ceph-noarch 1.8 M 2024-03-24T09:37:21.769 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-24T09:37:21.769 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-grafana-dashboards noarch 2:18.2.2-952.g738c2fd6.el9 ceph-noarch 23 k 2024-03-24T09:37:21.770 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-prometheus-alerts noarch 2:18.2.2-952.g738c2fd6.el9 ceph-noarch 15 k 2024-03-24T09:37:21.770 INFO:teuthology.orchestra.run.smithi155.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T09:37:21.770 INFO:teuthology.orchestra.run.smithi155.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T09:37:21.770 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:21.770 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-24T09:37:21.770 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-24T09:37:21.770 INFO:teuthology.orchestra.run.smithi155.stdout:Install 5 Packages 2024-03-24T09:37:21.770 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:21.771 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 2.0 M 2024-03-24T09:37:21.771 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 8.6 M 2024-03-24T09:37:21.771 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-24T09:37:22.214 INFO:teuthology.orchestra.run.smithi155.stdout:(1/5): ceph-grafana-dashboards-18.2.2-952.g738c 83 kB/s | 23 kB 00:00 2024-03-24T09:37:22.256 INFO:teuthology.orchestra.run.smithi155.stdout:(2/5): ceph-prometheus-alerts-18.2.2-952.g738c2 46 kB/s | 15 kB 00:00 2024-03-24T09:37:22.373 INFO:teuthology.orchestra.run.smithi155.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 194 kB/s | 31 kB 00:00 2024-03-24T09:37:22.456 INFO:teuthology.orchestra.run.smithi155.stdout:(4/5): ceph-mgr-dashboard-18.2.2-952.g738c2fd6. 3.4 MB/s | 1.8 MB 00:00 2024-03-24T09:37:22.482 INFO:teuthology.orchestra.run.smithi155.stdout:(5/5): python3-routes-2.5.1-5.el9.noarch.rpm 835 kB/s | 188 kB 00:00 2024-03-24T09:37:22.484 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:37:22.484 INFO:teuthology.orchestra.run.smithi155.stdout:Total 2.8 MB/s | 2.0 MB 00:00 2024-03-24T09:37:22.484 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-24T09:37:22.491 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-24T09:37:22.491 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-24T09:37:22.546 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-24T09:37:22.547 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-24T09:37:22.584 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-test-2:18.2.2-952.g738c2fd6.el9.x86_64 6/6 2024-03-24T09:37:22.584 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-test-2:18.2.2-952.g738c2fd6.el9.x86_64 1/6 2024-03-24T09:37:22.584 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libcephsqlite-2:18.2.2-952.g738c2fd6.el9.x86_64 2/6 2024-03-24T09:37:22.584 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-24T09:37:22.584 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-24T09:37:22.584 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-24T09:37:22.786 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-24T09:37:22.854 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-03-24T09:37:22.886 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-03-24T09:37:22.907 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-952.g738c2fd6.el9.no 3/5 2024-03-24T09:37:22.943 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-24T09:37:22.943 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:37:22.943 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-24T09:37:22.943 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-test-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:37:22.944 INFO:teuthology.orchestra.run.smithi022.stdout: jq-1.6-16.el9.x86_64 2024-03-24T09:37:22.944 INFO:teuthology.orchestra.run.smithi022.stdout: libcephsqlite-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:37:22.944 INFO:teuthology.orchestra.run.smithi022.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T09:37:22.944 INFO:teuthology.orchestra.run.smithi022.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T09:37:22.944 INFO:teuthology.orchestra.run.smithi022.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T09:37:22.944 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:37:22.944 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-24T09:37:23.138 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-952.g738c2fd6.el9.n 4/5 2024-03-24T09:37:23.168 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-952.g738c2fd6.el9.noarch 5/5 2024-03-24T09:37:23.223 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph 2024-03-24T09:37:23.668 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-952.g738c2fd6.el9.noarch 5/5 2024-03-24T09:37:23.668 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-952.g738c2fd6.el9.n 1/5 2024-03-24T09:37:23.669 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-952.g738c2fd6.el9.noarch 2/5 2024-03-24T09:37:23.669 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-952.g738c2fd6.el9.no 3/5 2024-03-24T09:37:23.669 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-03-24T09:37:23.793 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:00:38 ago on Sun 24 Mar 2024 09:36:45 AM UTC. 2024-03-24T09:37:23.953 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-24T09:37:23.955 INFO:teuthology.orchestra.run.smithi022.stdout:======================================================================================== 2024-03-24T09:37:23.955 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-24T09:37:23.955 INFO:teuthology.orchestra.run.smithi022.stdout:======================================================================================== 2024-03-24T09:37:23.955 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-24T09:37:23.955 INFO:teuthology.orchestra.run.smithi022.stdout: ceph x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 6.5 k 2024-03-24T09:37:23.955 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-24T09:37:23.955 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mds x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 2.1 M 2024-03-24T09:37:23.955 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 1.4 M 2024-03-24T09:37:23.956 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-modules-core noarch 2:18.2.2-952.g738c2fd6.el9 ceph-noarch 245 k 2024-03-24T09:37:23.956 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mon x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 4.7 M 2024-03-24T09:37:23.956 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-osd x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 16 M 2024-03-24T09:37:23.956 INFO:teuthology.orchestra.run.smithi022.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T09:37:23.956 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T09:37:23.956 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T09:37:23.956 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T09:37:23.956 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T09:37:23.956 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T09:37:23.956 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T09:37:23.956 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T09:37:23.956 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T09:37:23.956 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T09:37:23.957 INFO:teuthology.orchestra.run.smithi022.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T09:37:23.957 INFO:teuthology.orchestra.run.smithi022.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T09:37:23.957 INFO:teuthology.orchestra.run.smithi022.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T09:37:23.957 INFO:teuthology.orchestra.run.smithi022.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T09:37:23.957 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T09:37:23.957 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T09:37:23.957 INFO:teuthology.orchestra.run.smithi022.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T09:37:23.957 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T09:37:23.957 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T09:37:23.957 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T09:37:23.957 INFO:teuthology.orchestra.run.smithi022.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T09:37:23.957 INFO:teuthology.orchestra.run.smithi022.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T09:37:23.957 INFO:teuthology.orchestra.run.smithi022.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T09:37:23.957 INFO:teuthology.orchestra.run.smithi022.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T09:37:23.958 INFO:teuthology.orchestra.run.smithi022.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T09:37:23.958 INFO:teuthology.orchestra.run.smithi022.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T09:37:23.958 INFO:teuthology.orchestra.run.smithi022.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T09:37:23.958 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:37:23.958 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-24T09:37:23.958 INFO:teuthology.orchestra.run.smithi022.stdout:======================================================================================== 2024-03-24T09:37:23.958 INFO:teuthology.orchestra.run.smithi022.stdout:Install 33 Packages 2024-03-24T09:37:23.958 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:37:23.959 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-03-24T09:37:23.959 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:23.959 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-24T09:37:23.959 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-grafana-dashboards-2:18.2.2-952.g738c2fd6.el9.noarch 2024-03-24T09:37:23.959 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard-2:18.2.2-952.g738c2fd6.el9.noarch 2024-03-24T09:37:23.959 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-prometheus-alerts-2:18.2.2-952.g738c2fd6.el9.noarch 2024-03-24T09:37:23.959 INFO:teuthology.orchestra.run.smithi155.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T09:37:23.959 INFO:teuthology.orchestra.run.smithi155.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T09:37:23.959 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:23.959 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-24T09:37:23.960 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 29 M 2024-03-24T09:37:23.960 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 104 M 2024-03-24T09:37:23.960 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-24T09:37:24.199 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-24T09:37:24.296 INFO:teuthology.orchestra.run.smithi022.stdout:(1/33): ceph-18.2.2-952.g738c2fd6.el9.x86_64.rp 26 kB/s | 6.5 kB 00:00 2024-03-24T09:37:24.421 INFO:teuthology.orchestra.run.smithi022.stdout:(2/33): ceph-mds-18.2.2-952.g738c2fd6.el9.x86_6 5.6 MB/s | 2.1 MB 00:00 2024-03-24T09:37:24.755 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:59 ago on Sun 24 Mar 2024 09:36:25 AM UTC. 2024-03-24T09:37:24.904 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-24T09:37:24.905 INFO:teuthology.orchestra.run.smithi022.stdout:(3/33): ceph-mgr-18.2.2-952.g738c2fd6.el9.x86_6 1.7 MB/s | 1.4 MB 00:00 2024-03-24T09:37:24.905 INFO:teuthology.orchestra.run.smithi155.stdout:=========================================================================================== 2024-03-24T09:37:24.905 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-24T09:37:24.905 INFO:teuthology.orchestra.run.smithi155.stdout:=========================================================================================== 2024-03-24T09:37:24.906 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-24T09:37:24.906 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-952.g738c2fd6.el9 ceph-noarch 7.4 M 2024-03-24T09:37:24.906 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-24T09:37:24.906 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T09:37:24.906 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T09:37:24.906 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T09:37:24.906 INFO:teuthology.orchestra.run.smithi155.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T09:37:24.906 INFO:teuthology.orchestra.run.smithi155.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T09:37:24.906 INFO:teuthology.orchestra.run.smithi155.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T09:37:24.906 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T09:37:24.906 INFO:teuthology.orchestra.run.smithi155.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T09:37:24.906 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T09:37:24.906 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T09:37:24.906 INFO:teuthology.orchestra.run.smithi155.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T09:37:24.907 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:24.907 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-24T09:37:24.907 INFO:teuthology.orchestra.run.smithi155.stdout:=========================================================================================== 2024-03-24T09:37:24.907 INFO:teuthology.orchestra.run.smithi155.stdout:Install 12 Packages 2024-03-24T09:37:24.907 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:24.907 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 37 M 2024-03-24T09:37:24.907 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 209 M 2024-03-24T09:37:24.907 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-24T09:37:25.072 INFO:teuthology.orchestra.run.smithi022.stdout:(4/33): ceph-mgr-modules-core-18.2.2-952.g738c2 1.4 MB/s | 245 kB 00:00 2024-03-24T09:37:25.239 INFO:teuthology.orchestra.run.smithi022.stdout:(5/33): ceph-osd-18.2.2-952.g738c2fd6.el9.x86_6 20 MB/s | 16 MB 00:00 2024-03-24T09:37:25.305 INFO:teuthology.orchestra.run.smithi155.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 219 kB/s | 32 kB 00:00 2024-03-24T09:37:25.338 INFO:teuthology.orchestra.run.smithi155.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 536 kB/s | 17 kB 00:00 2024-03-24T09:37:25.356 INFO:teuthology.orchestra.run.smithi022.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 346 kB/s | 32 kB 00:00 2024-03-24T09:37:25.381 INFO:teuthology.orchestra.run.smithi155.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 841 kB/s | 34 kB 00:00 2024-03-24T09:37:25.381 INFO:teuthology.orchestra.run.smithi022.stdout:(7/33): python3-mako-1.1.4-6.el9.noarch.rpm 467 kB/s | 144 kB 00:00 2024-03-24T09:37:25.406 INFO:teuthology.orchestra.run.smithi022.stdout:(8/33): python3-pytz-2021.1-5.el9.noarch.rpm 944 kB/s | 47 kB 00:00 2024-03-24T09:37:25.431 INFO:teuthology.orchestra.run.smithi022.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 770 kB/s | 39 kB 00:00 2024-03-24T09:37:25.439 INFO:teuthology.orchestra.run.smithi155.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-24T09:37:25.465 INFO:teuthology.orchestra.run.smithi022.stdout:(10/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-24T09:37:25.532 INFO:teuthology.orchestra.run.smithi022.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 11 MB/s | 1.1 MB 00:00 2024-03-24T09:37:25.557 INFO:teuthology.orchestra.run.smithi022.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-03-24T09:37:25.583 INFO:teuthology.orchestra.run.smithi022.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-24T09:37:25.608 INFO:teuthology.orchestra.run.smithi022.stdout:(14/33): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-03-24T09:37:25.614 INFO:teuthology.orchestra.run.smithi155.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-03-24T09:37:25.690 INFO:teuthology.orchestra.run.smithi155.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-03-24T09:37:25.715 INFO:teuthology.orchestra.run.smithi155.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-03-24T09:37:25.758 INFO:teuthology.orchestra.run.smithi022.stdout:(15/33): python3-requests-2.25.1-8.el9.noarch.r 714 kB/s | 125 kB 00:00 2024-03-24T09:37:25.907 INFO:teuthology.orchestra.run.smithi155.stdout:(8/12): ceph-mgr-diskprediction-local-18.2.2-95 9.9 MB/s | 7.4 MB 00:00 2024-03-24T09:37:25.950 INFO:teuthology.orchestra.run.smithi022.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 127 kB/s | 43 kB 00:00 2024-03-24T09:37:26.074 INFO:teuthology.orchestra.run.smithi155.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-03-24T09:37:26.141 INFO:teuthology.orchestra.run.smithi155.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 845 kB/s | 195 kB 00:00 2024-03-24T09:37:26.191 INFO:teuthology.orchestra.run.smithi155.stdout:(11/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 803 kB 00:00 2024-03-24T09:37:26.234 INFO:teuthology.orchestra.run.smithi022.stdout:(17/33): ceph-mon-18.2.2-952.g738c2fd6.el9.x86_ 2.4 MB/s | 4.7 MB 00:01 2024-03-24T09:37:26.268 INFO:teuthology.orchestra.run.smithi022.stdout:(18/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 337 kB/s | 172 kB 00:00 2024-03-24T09:37:26.293 INFO:teuthology.orchestra.run.smithi022.stdout:(19/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.0 MB/s | 358 kB 00:00 2024-03-24T09:37:26.368 INFO:teuthology.orchestra.run.smithi022.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 176 kB/s | 18 kB 00:00 2024-03-24T09:37:26.394 INFO:teuthology.orchestra.run.smithi022.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 231 kB/s | 23 kB 00:00 2024-03-24T09:37:26.419 INFO:teuthology.orchestra.run.smithi022.stdout:(22/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 58 kB/s | 11 kB 00:00 2024-03-24T09:37:26.469 INFO:teuthology.orchestra.run.smithi022.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 193 kB/s | 19 kB 00:00 2024-03-24T09:37:26.494 INFO:teuthology.orchestra.run.smithi022.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 196 kB/s | 20 kB 00:00 2024-03-24T09:37:26.595 INFO:teuthology.orchestra.run.smithi022.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 630 kB/s | 79 kB 00:00 2024-03-24T09:37:26.620 INFO:teuthology.orchestra.run.smithi022.stdout:(26/33): python3-logutils-0.3.5-21.el9.noarch.r 230 kB/s | 46 kB 00:00 2024-03-24T09:37:26.645 INFO:teuthology.orchestra.run.smithi022.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-03-24T09:37:26.704 INFO:teuthology.orchestra.run.smithi022.stdout:(28/33): python3-portend-3.1.0-2.el9.noarch.rpm 151 kB/s | 16 kB 00:00 2024-03-24T09:37:26.708 INFO:teuthology.orchestra.run.smithi155.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 387 kB/s | 245 kB 00:00 2024-03-24T09:37:26.712 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:37:26.712 INFO:teuthology.orchestra.run.smithi155.stdout:Total 21 MB/s | 37 MB 00:01 2024-03-24T09:37:26.754 INFO:teuthology.orchestra.run.smithi022.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 330 kB/s | 36 kB 00:00 2024-03-24T09:37:26.754 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-24T09:37:26.778 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-24T09:37:26.778 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-24T09:37:26.829 INFO:teuthology.orchestra.run.smithi022.stdout:(30/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 430 kB/s | 90 kB 00:00 2024-03-24T09:37:26.888 INFO:teuthology.orchestra.run.smithi022.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.2 MB/s | 230 kB 00:00 2024-03-24T09:37:26.913 INFO:teuthology.orchestra.run.smithi022.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.6 MB/s | 427 kB 00:00 2024-03-24T09:37:26.930 INFO:teuthology.orchestra.run.smithi022.stdout:(33/33): python3-zc-lockfile-2.0-10.el9.noarch. 199 kB/s | 20 kB 00:00 2024-03-24T09:37:26.934 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:37:26.934 INFO:teuthology.orchestra.run.smithi022.stdout:Total 9.7 MB/s | 29 MB 00:02 2024-03-24T09:37:26.948 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-24T09:37:26.990 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-24T09:37:26.991 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-24T09:37:27.221 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-24T09:37:27.222 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-24T09:37:27.353 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-24T09:37:27.353 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-24T09:37:27.931 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-24T09:37:28.006 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-24T09:37:28.037 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-24T09:37:28.132 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-24T09:37:28.174 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-24T09:37:28.351 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-24T09:37:28.381 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-24T09:37:28.467 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-24T09:37:28.656 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-24T09:37:28.716 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-03-24T09:37:28.757 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-03-24T09:37:28.790 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-03-24T09:37:28.885 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-03-24T09:37:28.971 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-03-24T09:37:29.019 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-03-24T09:37:29.092 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-03-24T09:37:29.145 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-03-24T09:37:29.162 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-24T09:37:29.191 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-03-24T09:37:29.249 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-03-24T09:37:29.278 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-24T09:37:29.313 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-03-24T09:37:29.505 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-03-24T09:37:29.561 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-03-24T09:37:29.600 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-03-24T09:37:29.635 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-03-24T09:37:29.665 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-03-24T09:37:29.752 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-03-24T09:37:29.784 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-24T09:37:29.816 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-03-24T09:37:29.866 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-03-24T09:37:29.909 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-03-24T09:37:29.943 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-03-24T09:37:30.037 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-03-24T09:37:30.078 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-03-24T09:37:30.167 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-03-24T09:37:30.306 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-03-24T09:37:30.430 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-24T09:37:30.546 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-24T09:37:30.592 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-952.g738c2fd6.el9.n 28/33 2024-03-24T09:37:30.621 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-2:18.2.2-952.g738c2fd6.el9.x86_64 29/33 2024-03-24T09:37:30.645 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-2:18.2.2-952.g738c2fd6.el9.x86_64 29/33 2024-03-24T09:37:30.645 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:37:30.645 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T09:37:30.645 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T09:37:30.645 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T09:37:30.646 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:37:31.568 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-24T09:37:31.599 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-952.g738c2f 12/12 2024-03-24T09:37:32.148 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-osd-2:18.2.2-952.g738c2fd6.el9.x86_64 30/33 2024-03-24T09:37:32.175 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-osd-2:18.2.2-952.g738c2fd6.el9.x86_64 30/33 2024-03-24T09:37:32.175 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:37:32.175 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T09:37:32.175 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T09:37:32.175 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T09:37:32.175 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:37:32.648 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mon-2:18.2.2-952.g738c2fd6.el9.x86_64 31/33 2024-03-24T09:37:32.671 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mon-2:18.2.2-952.g738c2fd6.el9.x86_64 31/33 2024-03-24T09:37:32.671 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:37:32.671 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T09:37:32.671 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T09:37:32.671 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T09:37:32.671 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:37:32.891 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mds-2:18.2.2-952.g738c2fd6.el9.x86_64 32/33 2024-03-24T09:37:32.913 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mds-2:18.2.2-952.g738c2fd6.el9.x86_64 32/33 2024-03-24T09:37:32.914 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:37:32.914 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T09:37:32.914 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T09:37:32.914 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T09:37:32.914 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:37:32.982 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-2:18.2.2-952.g738c2fd6.el9.x86_64 33/33 2024-03-24T09:37:33.966 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-952.g738c2f 12/12 2024-03-24T09:37:33.967 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-952.g738c2f 1/12 2024-03-24T09:37:33.967 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-24T09:37:33.967 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-24T09:37:33.967 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-24T09:37:33.967 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-24T09:37:33.967 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-24T09:37:33.967 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-24T09:37:33.967 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-24T09:37:33.967 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-24T09:37:33.967 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-24T09:37:33.967 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-24T09:37:34.419 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-24T09:37:34.420 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:34.420 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-24T09:37:34.420 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-local-2:18.2.2-952.g738c2fd6.el9.noarch 2024-03-24T09:37:34.420 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T09:37:34.420 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T09:37:34.420 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T09:37:34.420 INFO:teuthology.orchestra.run.smithi155.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T09:37:34.420 INFO:teuthology.orchestra.run.smithi155.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T09:37:34.420 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T09:37:34.420 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T09:37:34.420 INFO:teuthology.orchestra.run.smithi155.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T09:37:34.420 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T09:37:34.420 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T09:37:34.420 INFO:teuthology.orchestra.run.smithi155.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T09:37:34.421 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:34.421 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-24T09:37:34.745 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-2:18.2.2-952.g738c2fd6.el9.x86_64 33/33 2024-03-24T09:37:34.745 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-2:18.2.2-952.g738c2fd6.el9.x86_64 1/33 2024-03-24T09:37:34.745 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mds-2:18.2.2-952.g738c2fd6.el9.x86_64 2/33 2024-03-24T09:37:34.745 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-2:18.2.2-952.g738c2fd6.el9.x86_64 3/33 2024-03-24T09:37:34.745 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mon-2:18.2.2-952.g738c2fd6.el9.x86_64 4/33 2024-03-24T09:37:34.745 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-osd-2:18.2.2-952.g738c2fd6.el9.x86_64 5/33 2024-03-24T09:37:34.745 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-952.g738c2fd6.el9.n 6/33 2024-03-24T09:37:34.746 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-03-24T09:37:34.746 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-03-24T09:37:34.746 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-03-24T09:37:34.746 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-03-24T09:37:34.746 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-03-24T09:37:34.746 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-03-24T09:37:34.746 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-03-24T09:37:34.746 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-03-24T09:37:34.746 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 15/33 2024-03-24T09:37:34.746 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/33 2024-03-24T09:37:34.746 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-03-24T09:37:34.746 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-24T09:37:34.746 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-03-24T09:37:34.746 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-03-24T09:37:34.747 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-03-24T09:37:34.747 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-03-24T09:37:34.747 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-03-24T09:37:34.747 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-03-24T09:37:34.747 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-03-24T09:37:34.747 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-03-24T09:37:34.747 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-24T09:37:34.747 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-03-24T09:37:34.747 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-03-24T09:37:34.747 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-03-24T09:37:34.747 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-03-24T09:37:34.747 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-03-24T09:37:34.772 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-rook 2024-03-24T09:37:35.339 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:10 ago on Sun 24 Mar 2024 09:36:25 AM UTC. 2024-03-24T09:37:35.488 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-24T09:37:35.489 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================= 2024-03-24T09:37:35.489 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-24T09:37:35.489 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================= 2024-03-24T09:37:35.489 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-24T09:37:35.490 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-rook noarch 2:18.2.2-952.g738c2fd6.el9 ceph-noarch 49 k 2024-03-24T09:37:35.490 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-24T09:37:35.490 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T09:37:35.490 INFO:teuthology.orchestra.run.smithi155.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T09:37:35.490 INFO:teuthology.orchestra.run.smithi155.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T09:37:35.490 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T09:37:35.490 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T09:37:35.490 INFO:teuthology.orchestra.run.smithi155.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T09:37:35.490 INFO:teuthology.orchestra.run.smithi155.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T09:37:35.490 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T09:37:35.490 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T09:37:35.490 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T09:37:35.490 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T09:37:35.491 INFO:teuthology.orchestra.run.smithi155.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T09:37:35.491 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:35.491 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-24T09:37:35.491 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================= 2024-03-24T09:37:35.491 INFO:teuthology.orchestra.run.smithi155.stdout:Install 13 Packages 2024-03-24T09:37:35.491 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:35.491 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 2.0 M 2024-03-24T09:37:35.492 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 26 M 2024-03-24T09:37:35.492 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-24T09:37:35.674 INFO:teuthology.orchestra.run.smithi155.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 174 kB/s | 24 kB 00:00 2024-03-24T09:37:35.700 INFO:teuthology.orchestra.run.smithi155.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 109 kB/s | 18 kB 00:00 2024-03-24T09:37:35.727 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-03-24T09:37:35.727 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:37:35.727 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-24T09:37:35.727 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:37:35.727 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mds-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:37:35.727 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:37:35.727 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-modules-core-2:18.2.2-952.g738c2fd6.el9.noarch 2024-03-24T09:37:35.727 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mon-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:37:35.727 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-osd-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:37:35.727 INFO:teuthology.orchestra.run.smithi022.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T09:37:35.728 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T09:37:35.728 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T09:37:35.728 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T09:37:35.728 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T09:37:35.728 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T09:37:35.728 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T09:37:35.728 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T09:37:35.728 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T09:37:35.728 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T09:37:35.728 INFO:teuthology.orchestra.run.smithi022.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T09:37:35.728 INFO:teuthology.orchestra.run.smithi022.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T09:37:35.728 INFO:teuthology.orchestra.run.smithi022.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T09:37:35.728 INFO:teuthology.orchestra.run.smithi022.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T09:37:35.728 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T09:37:35.728 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T09:37:35.728 INFO:teuthology.orchestra.run.smithi022.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T09:37:35.729 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T09:37:35.729 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T09:37:35.729 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T09:37:35.729 INFO:teuthology.orchestra.run.smithi022.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T09:37:35.729 INFO:teuthology.orchestra.run.smithi022.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T09:37:35.729 INFO:teuthology.orchestra.run.smithi022.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T09:37:35.729 INFO:teuthology.orchestra.run.smithi022.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T09:37:35.729 INFO:teuthology.orchestra.run.smithi022.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T09:37:35.729 INFO:teuthology.orchestra.run.smithi022.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T09:37:35.729 INFO:teuthology.orchestra.run.smithi022.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T09:37:35.729 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:37:35.729 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-24T09:37:35.741 INFO:teuthology.orchestra.run.smithi155.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-24T09:37:35.767 INFO:teuthology.orchestra.run.smithi155.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-24T09:37:35.792 INFO:teuthology.orchestra.run.smithi155.stdout:(5/13): ceph-mgr-rook-18.2.2-952.g738c2fd6.el9. 191 kB/s | 49 kB 00:00 2024-03-24T09:37:35.817 INFO:teuthology.orchestra.run.smithi155.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 211 kB 00:00 2024-03-24T09:37:35.843 INFO:teuthology.orchestra.run.smithi155.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 570 kB/s | 43 kB 00:00 2024-03-24T09:37:36.116 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-base 2024-03-24T09:37:36.118 INFO:teuthology.orchestra.run.smithi155.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 50 kB/s | 15 kB 00:00 2024-03-24T09:37:36.143 INFO:teuthology.orchestra.run.smithi155.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 99 kB/s | 32 kB 00:00 2024-03-24T09:37:36.269 INFO:teuthology.orchestra.run.smithi155.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 476 kB/s | 203 kB 00:00 2024-03-24T09:37:36.294 INFO:teuthology.orchestra.run.smithi155.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 393 kB/s | 59 kB 00:00 2024-03-24T09:37:36.377 INFO:teuthology.orchestra.run.smithi155.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 826 kB/s | 90 kB 00:00 2024-03-24T09:37:36.461 INFO:teuthology.orchestra.run.smithi155.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.0 MB/s | 1.0 MB 00:00 2024-03-24T09:37:36.465 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:37:36.465 INFO:teuthology.orchestra.run.smithi155.stdout:Total 2.1 MB/s | 2.0 MB 00:00 2024-03-24T09:37:36.466 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-24T09:37:36.479 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-24T09:37:36.479 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-24T09:37:36.569 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-24T09:37:36.569 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-24T09:37:36.694 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:00:51 ago on Sun 24 Mar 2024 09:36:45 AM UTC. 2024-03-24T09:37:36.794 INFO:teuthology.orchestra.run.smithi022.stdout:Package ceph-base-2:18.2.2-952.g738c2fd6.el9.x86_64 is already installed. 2024-03-24T09:37:36.849 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-24T09:37:36.850 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2024-03-24T09:37:36.851 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-24T09:37:36.914 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install cephadm 2024-03-24T09:37:37.076 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-24T09:37:37.250 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-24T09:37:37.305 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-24T09:37:37.360 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-24T09:37:37.397 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-24T09:37:37.439 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-24T09:37:37.484 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:00:52 ago on Sun 24 Mar 2024 09:36:45 AM UTC. 2024-03-24T09:37:37.534 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-24T09:37:37.638 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-24T09:37:37.639 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T09:37:37.639 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-24T09:37:37.639 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T09:37:37.639 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-24T09:37:37.639 INFO:teuthology.orchestra.run.smithi022.stdout: cephadm noarch 2:18.2.2-952.g738c2fd6.el9 ceph-noarch 224 k 2024-03-24T09:37:37.639 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:37:37.639 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-24T09:37:37.639 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T09:37:37.639 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-24T09:37:37.640 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:37:37.640 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 224 k 2024-03-24T09:37:37.640 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 215 k 2024-03-24T09:37:37.640 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-24T09:37:37.643 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-24T09:37:37.690 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-24T09:37:37.865 INFO:teuthology.orchestra.run.smithi022.stdout:cephadm-18.2.2-952.g738c2fd6.el9.noarch.rpm 994 kB/s | 224 kB 00:00 2024-03-24T09:37:37.866 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:37:37.866 INFO:teuthology.orchestra.run.smithi022.stdout:Total 987 kB/s | 224 kB 00:00 2024-03-24T09:37:37.866 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-24T09:37:37.872 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-24T09:37:37.872 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-24T09:37:37.875 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-24T09:37:37.875 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-24T09:37:37.950 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-24T09:37:38.359 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: cephadm-2:18.2.2-952.g738c2fd6.el9.noarch 1/1 2024-03-24T09:37:38.421 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-24T09:37:38.455 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : cephadm-2:18.2.2-952.g738c2fd6.el9.noarch 1/1 2024-03-24T09:37:38.470 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-24T09:37:38.504 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-24T09:37:38.552 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-24T09:37:38.574 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-rook-2:18.2.2-952.g738c2fd6.el9.noarch 13/13 2024-03-24T09:37:38.823 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: cephadm-2:18.2.2-952.g738c2fd6.el9.noarch 1/1 2024-03-24T09:37:39.107 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : cephadm-2:18.2.2-952.g738c2fd6.el9.noarch 1/1 2024-03-24T09:37:39.108 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:37:39.108 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-24T09:37:39.108 INFO:teuthology.orchestra.run.smithi022.stdout: cephadm-2:18.2.2-952.g738c2fd6.el9.noarch 2024-03-24T09:37:39.108 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:37:39.108 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-24T09:37:39.327 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-immutable-object-cache 2024-03-24T09:37:39.348 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-952.g738c2fd6.el9.noarch 13/13 2024-03-24T09:37:39.348 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-rook-2:18.2.2-952.g738c2fd6.el9.noarch 1/13 2024-03-24T09:37:39.348 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-24T09:37:39.348 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-24T09:37:39.348 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-24T09:37:39.348 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-24T09:37:39.348 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-24T09:37:39.348 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-24T09:37:39.348 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-24T09:37:39.348 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-24T09:37:39.348 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-24T09:37:39.348 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-24T09:37:39.348 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-24T09:37:39.707 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-24T09:37:39.707 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:39.707 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-24T09:37:39.707 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-rook-2:18.2.2-952.g738c2fd6.el9.noarch 2024-03-24T09:37:39.707 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T09:37:39.707 INFO:teuthology.orchestra.run.smithi155.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T09:37:39.707 INFO:teuthology.orchestra.run.smithi155.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T09:37:39.707 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T09:37:39.708 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T09:37:39.708 INFO:teuthology.orchestra.run.smithi155.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T09:37:39.708 INFO:teuthology.orchestra.run.smithi155.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T09:37:39.708 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T09:37:39.708 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T09:37:39.708 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T09:37:39.708 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T09:37:39.708 INFO:teuthology.orchestra.run.smithi155.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T09:37:39.708 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:39.708 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-24T09:37:39.900 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:00:54 ago on Sun 24 Mar 2024 09:36:45 AM UTC. 2024-03-24T09:37:39.943 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-cephadm 2024-03-24T09:37:40.056 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-24T09:37:40.057 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T09:37:40.057 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2024-03-24T09:37:40.058 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T09:37:40.058 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-24T09:37:40.058 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 143 k 2024-03-24T09:37:40.058 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:37:40.058 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-24T09:37:40.058 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T09:37:40.058 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-24T09:37:40.058 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:37:40.058 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 143 k 2024-03-24T09:37:40.058 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 395 k 2024-03-24T09:37:40.058 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-24T09:37:40.280 INFO:teuthology.orchestra.run.smithi022.stdout:ceph-immutable-object-cache-18.2.2-952.g738c2fd 643 kB/s | 143 kB 00:00 2024-03-24T09:37:40.281 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:37:40.281 INFO:teuthology.orchestra.run.smithi022.stdout:Total 639 kB/s | 143 kB 00:00 2024-03-24T09:37:40.281 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-24T09:37:40.289 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-24T09:37:40.289 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-24T09:37:40.344 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-24T09:37:40.344 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-24T09:37:40.486 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-24T09:37:40.500 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:15 ago on Sun 24 Mar 2024 09:36:25 AM UTC. 2024-03-24T09:37:40.556 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-952.g738c2fd6.e 1/1 2024-03-24T09:37:40.580 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-952.g738c2fd6.e 1/1 2024-03-24T09:37:40.580 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:37:40.580 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T09:37:40.580 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:37:40.650 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-24T09:37:40.651 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================= 2024-03-24T09:37:40.651 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-24T09:37:40.651 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================= 2024-03-24T09:37:40.651 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-24T09:37:40.651 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-cephadm noarch 2:18.2.2-952.g738c2fd6.el9 ceph-noarch 138 k 2024-03-24T09:37:40.651 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-24T09:37:40.651 INFO:teuthology.orchestra.run.smithi155.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T09:37:40.651 INFO:teuthology.orchestra.run.smithi155.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T09:37:40.651 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T09:37:40.652 INFO:teuthology.orchestra.run.smithi155.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T09:37:40.652 INFO:teuthology.orchestra.run.smithi155.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T09:37:40.652 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:40.652 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-24T09:37:40.652 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================= 2024-03-24T09:37:40.652 INFO:teuthology.orchestra.run.smithi155.stdout:Install 6 Packages 2024-03-24T09:37:40.652 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:40.653 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 6.8 M 2024-03-24T09:37:40.653 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 33 M 2024-03-24T09:37:40.653 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-24T09:37:41.072 INFO:teuthology.orchestra.run.smithi155.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-24T09:37:41.222 INFO:teuthology.orchestra.run.smithi155.stdout:(2/6): ceph-mgr-cephadm-18.2.2-952.g738c2fd6.el 334 kB/s | 138 kB 00:00 2024-03-24T09:37:41.546 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-952.g738c2fd6.e 1/1 2024-03-24T09:37:41.547 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:37:41.547 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-24T09:37:41.547 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-immutable-object-cache-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:37:41.547 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:37:41.547 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-24T09:37:41.690 INFO:teuthology.orchestra.run.smithi155.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 123 kB/s | 58 kB 00:00 2024-03-24T09:37:41.809 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-mgr 2024-03-24T09:37:41.823 INFO:teuthology.orchestra.run.smithi155.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 730 kB/s | 548 kB 00:00 2024-03-24T09:37:41.848 INFO:teuthology.orchestra.run.smithi155.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 322 kB/s | 51 kB 00:00 2024-03-24T09:37:41.932 INFO:teuthology.orchestra.run.smithi155.stdout:(6/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 221 kB/s | 248 kB 00:01 2024-03-24T09:37:41.935 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:37:41.935 INFO:teuthology.orchestra.run.smithi155.stdout:Total 5.3 MB/s | 6.8 MB 00:01 2024-03-24T09:37:41.948 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-24T09:37:41.958 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-24T09:37:41.958 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-24T09:37:42.049 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-24T09:37:42.049 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-24T09:37:42.360 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-24T09:37:42.384 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:00:57 ago on Sun 24 Mar 2024 09:36:45 AM UTC. 2024-03-24T09:37:42.483 INFO:teuthology.orchestra.run.smithi022.stdout:Package ceph-mgr-2:18.2.2-952.g738c2fd6.el9.x86_64 is already installed. 2024-03-24T09:37:42.537 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-24T09:37:42.538 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2024-03-24T09:37:42.539 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-24T09:37:42.539 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-24T09:37:42.601 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-24T09:37:42.603 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-mgr-dashboard 2024-03-24T09:37:43.115 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-24T09:37:43.169 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:00:58 ago on Sun 24 Mar 2024 09:36:45 AM UTC. 2024-03-24T09:37:43.203 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-24T09:37:43.265 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-24T09:37:43.288 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-952.g738c2fd6.el9.noarch 6/6 2024-03-24T09:37:43.323 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-24T09:37:43.324 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T09:37:43.324 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-24T09:37:43.324 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T09:37:43.325 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-24T09:37:43.325 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-dashboard noarch 2:18.2.2-952.g738c2fd6.el9 ceph-noarch 1.8 M 2024-03-24T09:37:43.325 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-24T09:37:43.325 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-grafana-dashboards noarch 2:18.2.2-952.g738c2fd6.el9 ceph-noarch 23 k 2024-03-24T09:37:43.325 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-prometheus-alerts noarch 2:18.2.2-952.g738c2fd6.el9 ceph-noarch 15 k 2024-03-24T09:37:43.325 INFO:teuthology.orchestra.run.smithi022.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T09:37:43.325 INFO:teuthology.orchestra.run.smithi022.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T09:37:43.325 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:37:43.325 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-24T09:37:43.325 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T09:37:43.325 INFO:teuthology.orchestra.run.smithi022.stdout:Install 5 Packages 2024-03-24T09:37:43.325 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:37:43.326 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 2.0 M 2024-03-24T09:37:43.326 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 8.6 M 2024-03-24T09:37:43.326 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-24T09:37:43.567 INFO:teuthology.orchestra.run.smithi022.stdout:(1/5): ceph-prometheus-alerts-18.2.2-952.g738c2 76 kB/s | 15 kB 00:00 2024-03-24T09:37:43.609 INFO:teuthology.orchestra.run.smithi022.stdout:(2/5): ceph-grafana-dashboards-18.2.2-952.g738c 98 kB/s | 23 kB 00:00 2024-03-24T09:37:43.759 INFO:teuthology.orchestra.run.smithi022.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 160 kB/s | 31 kB 00:00 2024-03-24T09:37:43.851 INFO:teuthology.orchestra.run.smithi022.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 779 kB/s | 188 kB 00:00 2024-03-24T09:37:44.076 INFO:teuthology.orchestra.run.smithi022.stdout:(5/5): ceph-mgr-dashboard-18.2.2-952.g738c2fd6. 2.5 MB/s | 1.8 MB 00:00 2024-03-24T09:37:44.079 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:37:44.080 INFO:teuthology.orchestra.run.smithi022.stdout:Total 2.7 MB/s | 2.0 MB 00:00 2024-03-24T09:37:44.080 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-24T09:37:44.087 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-24T09:37:44.087 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-24T09:37:44.113 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-952.g738c2fd6.el9.noarch 6/6 2024-03-24T09:37:44.113 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-952.g738c2fd6.el9.noarch 1/6 2024-03-24T09:37:44.113 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-24T09:37:44.113 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-24T09:37:44.113 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-24T09:37:44.113 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-24T09:37:44.145 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-24T09:37:44.146 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-24T09:37:44.387 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-24T09:37:44.414 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-24T09:37:44.415 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:44.415 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-24T09:37:44.415 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-cephadm-2:18.2.2-952.g738c2fd6.el9.noarch 2024-03-24T09:37:44.415 INFO:teuthology.orchestra.run.smithi155.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T09:37:44.415 INFO:teuthology.orchestra.run.smithi155.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T09:37:44.415 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T09:37:44.415 INFO:teuthology.orchestra.run.smithi155.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T09:37:44.415 INFO:teuthology.orchestra.run.smithi155.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T09:37:44.415 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:44.415 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-24T09:37:44.483 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-03-24T09:37:44.523 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-03-24T09:37:44.552 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-952.g738c2fd6.el9.no 3/5 2024-03-24T09:37:44.660 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-fuse 2024-03-24T09:37:44.801 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-952.g738c2fd6.el9.n 4/5 2024-03-24T09:37:44.838 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-952.g738c2fd6.el9.noarch 5/5 2024-03-24T09:37:45.221 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:20 ago on Sun 24 Mar 2024 09:36:25 AM UTC. 2024-03-24T09:37:45.370 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-24T09:37:45.371 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-24T09:37:45.371 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-24T09:37:45.371 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-24T09:37:45.371 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-24T09:37:45.371 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 844 k 2024-03-24T09:37:45.371 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-24T09:37:45.371 INFO:teuthology.orchestra.run.smithi155.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T09:37:45.371 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:45.371 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-24T09:37:45.372 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-24T09:37:45.372 INFO:teuthology.orchestra.run.smithi155.stdout:Install 2 Packages 2024-03-24T09:37:45.372 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:45.372 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 922 k 2024-03-24T09:37:45.372 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 2.7 M 2024-03-24T09:37:45.372 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-24T09:37:45.400 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-952.g738c2fd6.el9.noarch 5/5 2024-03-24T09:37:45.400 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-952.g738c2fd6.el9.n 1/5 2024-03-24T09:37:45.400 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-952.g738c2fd6.el9.noarch 2/5 2024-03-24T09:37:45.400 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-952.g738c2fd6.el9.no 3/5 2024-03-24T09:37:45.400 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-03-24T09:37:45.502 INFO:teuthology.orchestra.run.smithi155.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 607 kB/s | 79 kB 00:00 2024-03-24T09:37:45.684 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-03-24T09:37:45.684 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:37:45.685 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-24T09:37:45.685 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-grafana-dashboards-2:18.2.2-952.g738c2fd6.el9.noarch 2024-03-24T09:37:45.685 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-dashboard-2:18.2.2-952.g738c2fd6.el9.noarch 2024-03-24T09:37:45.685 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-prometheus-alerts-2:18.2.2-952.g738c2fd6.el9.noarch 2024-03-24T09:37:45.685 INFO:teuthology.orchestra.run.smithi022.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T09:37:45.685 INFO:teuthology.orchestra.run.smithi022.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T09:37:45.685 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:37:45.685 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-24T09:37:45.726 INFO:teuthology.orchestra.run.smithi155.stdout:(2/2): ceph-fuse-18.2.2-952.g738c2fd6.el9.x86_6 2.3 MB/s | 844 kB 00:00 2024-03-24T09:37:45.728 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:37:45.728 INFO:teuthology.orchestra.run.smithi155.stdout:Total 2.5 MB/s | 922 kB 00:00 2024-03-24T09:37:45.728 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-24T09:37:45.748 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-24T09:37:45.748 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-24T09:37:45.819 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-24T09:37:45.819 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-24T09:37:45.920 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-24T09:37:45.991 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-24T09:37:46.140 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-24T09:37:46.210 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-fuse-2:18.2.2-952.g738c2fd6.el9.x86_64 2/2 2024-03-24T09:37:46.488 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:01 ago on Sun 24 Mar 2024 09:36:45 AM UTC. 2024-03-24T09:37:46.644 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-24T09:37:46.645 INFO:teuthology.orchestra.run.smithi022.stdout:=========================================================================================== 2024-03-24T09:37:46.645 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-24T09:37:46.645 INFO:teuthology.orchestra.run.smithi022.stdout:=========================================================================================== 2024-03-24T09:37:46.645 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-24T09:37:46.645 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-952.g738c2fd6.el9 ceph-noarch 7.4 M 2024-03-24T09:37:46.646 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-24T09:37:46.646 INFO:teuthology.orchestra.run.smithi022.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T09:37:46.646 INFO:teuthology.orchestra.run.smithi022.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T09:37:46.646 INFO:teuthology.orchestra.run.smithi022.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T09:37:46.646 INFO:teuthology.orchestra.run.smithi022.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T09:37:46.646 INFO:teuthology.orchestra.run.smithi022.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T09:37:46.646 INFO:teuthology.orchestra.run.smithi022.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T09:37:46.646 INFO:teuthology.orchestra.run.smithi022.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T09:37:46.646 INFO:teuthology.orchestra.run.smithi022.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T09:37:46.646 INFO:teuthology.orchestra.run.smithi022.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T09:37:46.646 INFO:teuthology.orchestra.run.smithi022.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T09:37:46.647 INFO:teuthology.orchestra.run.smithi022.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T09:37:46.647 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:37:46.647 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-24T09:37:46.647 INFO:teuthology.orchestra.run.smithi022.stdout:=========================================================================================== 2024-03-24T09:37:46.647 INFO:teuthology.orchestra.run.smithi022.stdout:Install 12 Packages 2024-03-24T09:37:46.647 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:37:46.648 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 37 M 2024-03-24T09:37:46.648 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 209 M 2024-03-24T09:37:46.648 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-24T09:37:46.837 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-fuse-2:18.2.2-952.g738c2fd6.el9.x86_64 2/2 2024-03-24T09:37:46.837 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-fuse-2:18.2.2-952.g738c2fd6.el9.x86_64 1/2 2024-03-24T09:37:46.916 INFO:teuthology.orchestra.run.smithi022.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 226 kB/s | 32 kB 00:00 2024-03-24T09:37:46.941 INFO:teuthology.orchestra.run.smithi022.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 725 kB/s | 17 kB 00:00 2024-03-24T09:37:46.975 INFO:teuthology.orchestra.run.smithi022.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-24T09:37:47.034 INFO:teuthology.orchestra.run.smithi022.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-03-24T09:37:47.147 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-24T09:37:47.148 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:47.148 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-24T09:37:47.148 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse-2:18.2.2-952.g738c2fd6.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-24T09:37:47.148 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:47.148 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-24T09:37:47.209 INFO:teuthology.orchestra.run.smithi022.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-03-24T09:37:47.276 INFO:teuthology.orchestra.run.smithi022.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-24T09:37:47.301 INFO:teuthology.orchestra.run.smithi022.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-24T09:37:47.398 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install librados-devel 2024-03-24T09:37:47.585 INFO:teuthology.orchestra.run.smithi022.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 51 MB/s | 16 MB 00:00 2024-03-24T09:37:47.710 INFO:teuthology.orchestra.run.smithi022.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.9 MB/s | 803 kB 00:00 2024-03-24T09:37:47.861 INFO:teuthology.orchestra.run.smithi022.stdout:(10/12): ceph-mgr-diskprediction-local-18.2.2-9 6.8 MB/s | 7.4 MB 00:01 2024-03-24T09:37:47.886 INFO:teuthology.orchestra.run.smithi022.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 650 kB/s | 195 kB 00:00 2024-03-24T09:37:47.968 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:22 ago on Sun 24 Mar 2024 09:36:25 AM UTC. 2024-03-24T09:37:48.124 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-24T09:37:48.125 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-24T09:37:48.125 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-03-24T09:37:48.125 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-24T09:37:48.125 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-24T09:37:48.125 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 127 k 2024-03-24T09:37:48.125 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:48.125 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-24T09:37:48.125 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-24T09:37:48.125 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-24T09:37:48.125 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:48.126 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 127 k 2024-03-24T09:37:48.126 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 456 k 2024-03-24T09:37:48.126 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-24T09:37:48.357 INFO:teuthology.orchestra.run.smithi155.stdout:librados-devel-18.2.2-952.g738c2fd6.el9.x86_64. 548 kB/s | 127 kB 00:00 2024-03-24T09:37:48.357 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:37:48.358 INFO:teuthology.orchestra.run.smithi155.stdout:Total 544 kB/s | 127 kB 00:00 2024-03-24T09:37:48.358 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-24T09:37:48.361 INFO:teuthology.orchestra.run.smithi022.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 377 kB/s | 245 kB 00:00 2024-03-24T09:37:48.363 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-24T09:37:48.363 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-24T09:37:48.364 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:37:48.364 INFO:teuthology.orchestra.run.smithi022.stdout:Total 22 MB/s | 37 MB 00:01 2024-03-24T09:37:48.407 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-24T09:37:48.417 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-24T09:37:48.418 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-24T09:37:48.432 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-24T09:37:48.433 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-24T09:37:48.575 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-24T09:37:48.680 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librados-devel-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T09:37:48.876 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-24T09:37:48.876 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-24T09:37:49.105 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados-devel-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T09:37:49.370 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados-devel-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T09:37:49.371 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:49.371 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-24T09:37:49.371 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:37:49.371 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:49.371 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-24T09:37:49.597 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-24T09:37:49.601 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install libcephfs2 2024-03-24T09:37:49.676 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-24T09:37:49.715 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-24T09:37:49.814 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-24T09:37:49.852 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-24T09:37:50.035 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-24T09:37:50.076 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-24T09:37:50.158 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-24T09:37:50.168 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:25 ago on Sun 24 Mar 2024 09:36:25 AM UTC. 2024-03-24T09:37:50.266 INFO:teuthology.orchestra.run.smithi155.stdout:Package libcephfs2-2:18.2.2-952.g738c2fd6.el9.x86_64 is already installed. 2024-03-24T09:37:50.322 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-24T09:37:50.323 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-03-24T09:37:50.323 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-24T09:37:50.387 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install libcephfs-devel 2024-03-24T09:37:50.853 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-24T09:37:50.950 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:25 ago on Sun 24 Mar 2024 09:36:25 AM UTC. 2024-03-24T09:37:50.967 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-24T09:37:51.099 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-24T09:37:51.100 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-24T09:37:51.100 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-03-24T09:37:51.100 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-24T09:37:51.100 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-24T09:37:51.100 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-devel x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 31 k 2024-03-24T09:37:51.100 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:51.101 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-24T09:37:51.101 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-24T09:37:51.101 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-24T09:37:51.101 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:51.101 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 31 k 2024-03-24T09:37:51.101 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 139 k 2024-03-24T09:37:51.101 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-24T09:37:51.269 INFO:teuthology.orchestra.run.smithi155.stdout:libcephfs-devel-18.2.2-952.g738c2fd6.el9.x86_64 184 kB/s | 31 kB 00:00 2024-03-24T09:37:51.270 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:37:51.270 INFO:teuthology.orchestra.run.smithi155.stdout:Total 183 kB/s | 31 kB 00:00 2024-03-24T09:37:51.271 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-24T09:37:51.274 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-24T09:37:51.274 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-24T09:37:51.307 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-24T09:37:51.307 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-24T09:37:51.392 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-24T09:37:51.493 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephfs-devel-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T09:37:51.876 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T09:37:52.120 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephfs-devel-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T09:37:52.120 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:52.120 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-24T09:37:52.120 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-devel-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:37:52.120 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:52.120 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-24T09:37:52.272 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-24T09:37:52.313 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install librados2 2024-03-24T09:37:52.885 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:27 ago on Sun 24 Mar 2024 09:36:25 AM UTC. 2024-03-24T09:37:52.983 INFO:teuthology.orchestra.run.smithi155.stdout:Package librados2-2:18.2.2-952.g738c2fd6.el9.x86_64 is already installed. 2024-03-24T09:37:53.035 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-24T09:37:53.036 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-03-24T09:37:53.036 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-24T09:37:53.098 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install librbd1 2024-03-24T09:37:53.335 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-24T09:37:53.400 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-952.g738c2f 12/12 2024-03-24T09:37:53.658 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:28 ago on Sun 24 Mar 2024 09:36:25 AM UTC. 2024-03-24T09:37:53.756 INFO:teuthology.orchestra.run.smithi155.stdout:Package librbd1-2:18.2.2-952.g738c2fd6.el9.x86_64 is already installed. 2024-03-24T09:37:53.808 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-24T09:37:53.809 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-03-24T09:37:53.809 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-24T09:37:53.872 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-rados 2024-03-24T09:37:54.439 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:29 ago on Sun 24 Mar 2024 09:36:25 AM UTC. 2024-03-24T09:37:54.538 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-rados-2:18.2.2-952.g738c2fd6.el9.x86_64 is already installed. 2024-03-24T09:37:54.590 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-24T09:37:54.591 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-03-24T09:37:54.591 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-24T09:37:54.656 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-rgw 2024-03-24T09:37:55.220 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:30 ago on Sun 24 Mar 2024 09:36:25 AM UTC. 2024-03-24T09:37:55.316 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-rgw-2:18.2.2-952.g738c2fd6.el9.x86_64 is already installed. 2024-03-24T09:37:55.371 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-24T09:37:55.372 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-03-24T09:37:55.372 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-24T09:37:55.443 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-cephfs 2024-03-24T09:37:56.006 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:31 ago on Sun 24 Mar 2024 09:36:25 AM UTC. 2024-03-24T09:37:56.023 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-952.g738c2f 12/12 2024-03-24T09:37:56.023 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-952.g738c2f 1/12 2024-03-24T09:37:56.023 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-24T09:37:56.023 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-24T09:37:56.023 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-24T09:37:56.024 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-24T09:37:56.024 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-24T09:37:56.024 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-24T09:37:56.024 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-24T09:37:56.024 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-24T09:37:56.024 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-24T09:37:56.024 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-24T09:37:56.105 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-cephfs-2:18.2.2-952.g738c2fd6.el9.x86_64 is already installed. 2024-03-24T09:37:56.157 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-24T09:37:56.158 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-03-24T09:37:56.158 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-24T09:37:56.222 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-rbd 2024-03-24T09:37:56.451 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-24T09:37:56.451 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:37:56.451 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-24T09:37:56.451 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-diskprediction-local-2:18.2.2-952.g738c2fd6.el9.noarch 2024-03-24T09:37:56.451 INFO:teuthology.orchestra.run.smithi022.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T09:37:56.451 INFO:teuthology.orchestra.run.smithi022.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T09:37:56.451 INFO:teuthology.orchestra.run.smithi022.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T09:37:56.451 INFO:teuthology.orchestra.run.smithi022.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T09:37:56.451 INFO:teuthology.orchestra.run.smithi022.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T09:37:56.451 INFO:teuthology.orchestra.run.smithi022.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T09:37:56.451 INFO:teuthology.orchestra.run.smithi022.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T09:37:56.452 INFO:teuthology.orchestra.run.smithi022.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T09:37:56.452 INFO:teuthology.orchestra.run.smithi022.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T09:37:56.452 INFO:teuthology.orchestra.run.smithi022.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T09:37:56.452 INFO:teuthology.orchestra.run.smithi022.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T09:37:56.452 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:37:56.452 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-24T09:37:56.782 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:31 ago on Sun 24 Mar 2024 09:36:25 AM UTC. 2024-03-24T09:37:56.829 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-mgr-rook 2024-03-24T09:37:56.878 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-rbd-2:18.2.2-952.g738c2fd6.el9.x86_64 is already installed. 2024-03-24T09:37:56.931 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-24T09:37:56.932 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-03-24T09:37:56.932 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-24T09:37:56.997 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install rbd-fuse 2024-03-24T09:37:57.400 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:12 ago on Sun 24 Mar 2024 09:36:45 AM UTC. 2024-03-24T09:37:57.556 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-24T09:37:57.558 INFO:teuthology.orchestra.run.smithi022.stdout:======================================================================================= 2024-03-24T09:37:57.558 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-24T09:37:57.558 INFO:teuthology.orchestra.run.smithi022.stdout:======================================================================================= 2024-03-24T09:37:57.558 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-24T09:37:57.558 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-rook noarch 2:18.2.2-952.g738c2fd6.el9 ceph-noarch 49 k 2024-03-24T09:37:57.558 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-24T09:37:57.558 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T09:37:57.558 INFO:teuthology.orchestra.run.smithi022.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T09:37:57.558 INFO:teuthology.orchestra.run.smithi022.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T09:37:57.559 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T09:37:57.559 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T09:37:57.559 INFO:teuthology.orchestra.run.smithi022.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T09:37:57.559 INFO:teuthology.orchestra.run.smithi022.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T09:37:57.559 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T09:37:57.559 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T09:37:57.559 INFO:teuthology.orchestra.run.smithi022.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T09:37:57.559 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T09:37:57.559 INFO:teuthology.orchestra.run.smithi022.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T09:37:57.559 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:37:57.559 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-24T09:37:57.559 INFO:teuthology.orchestra.run.smithi022.stdout:======================================================================================= 2024-03-24T09:37:57.559 INFO:teuthology.orchestra.run.smithi022.stdout:Install 13 Packages 2024-03-24T09:37:57.559 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:37:57.560 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 2.0 M 2024-03-24T09:37:57.560 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 26 M 2024-03-24T09:37:57.560 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-24T09:37:57.640 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:32 ago on Sun 24 Mar 2024 09:36:25 AM UTC. 2024-03-24T09:37:57.734 INFO:teuthology.orchestra.run.smithi022.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 138 kB/s | 18 kB 00:00 2024-03-24T09:37:57.759 INFO:teuthology.orchestra.run.smithi022.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 156 kB/s | 24 kB 00:00 2024-03-24T09:37:57.789 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-24T09:37:57.790 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-24T09:37:57.790 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2024-03-24T09:37:57.790 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-24T09:37:57.790 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-24T09:37:57.790 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 87 k 2024-03-24T09:37:57.790 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:57.790 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-24T09:37:57.790 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-24T09:37:57.790 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-24T09:37:57.790 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:57.791 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 87 k 2024-03-24T09:37:57.791 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 227 k 2024-03-24T09:37:57.791 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-24T09:37:57.818 INFO:teuthology.orchestra.run.smithi022.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 174 kB 00:00 2024-03-24T09:37:57.851 INFO:teuthology.orchestra.run.smithi022.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 132 kB 00:00 2024-03-24T09:37:57.868 INFO:teuthology.orchestra.run.smithi022.stdout:(5/13): ceph-mgr-rook-18.2.2-952.g738c2fd6.el9. 185 kB/s | 49 kB 00:00 2024-03-24T09:37:57.894 INFO:teuthology.orchestra.run.smithi022.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 211 kB 00:00 2024-03-24T09:37:57.910 INFO:teuthology.orchestra.run.smithi022.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 733 kB/s | 43 kB 00:00 2024-03-24T09:37:58.115 INFO:teuthology.orchestra.run.smithi155.stdout:rbd-fuse-18.2.2-952.g738c2fd6.el9.x86_64.rpm 267 kB/s | 87 kB 00:00 2024-03-24T09:37:58.115 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:37:58.116 INFO:teuthology.orchestra.run.smithi155.stdout:Total 266 kB/s | 87 kB 00:00 2024-03-24T09:37:58.116 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-24T09:37:58.123 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-24T09:37:58.123 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-24T09:37:58.169 INFO:teuthology.orchestra.run.smithi022.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 54 kB/s | 15 kB 00:00 2024-03-24T09:37:58.178 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-24T09:37:58.178 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-24T09:37:58.186 INFO:teuthology.orchestra.run.smithi022.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 110 kB/s | 32 kB 00:00 2024-03-24T09:37:58.303 INFO:teuthology.orchestra.run.smithi022.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 507 kB/s | 59 kB 00:00 2024-03-24T09:37:58.311 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-24T09:37:58.328 INFO:teuthology.orchestra.run.smithi022.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 485 kB/s | 203 kB 00:00 2024-03-24T09:37:58.413 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-fuse-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T09:37:58.428 INFO:teuthology.orchestra.run.smithi022.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 716 kB/s | 90 kB 00:00 2024-03-24T09:37:58.512 INFO:teuthology.orchestra.run.smithi022.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.0 MB/s | 1.0 MB 00:00 2024-03-24T09:37:58.514 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:37:58.515 INFO:teuthology.orchestra.run.smithi022.stdout:Total 2.1 MB/s | 2.0 MB 00:00 2024-03-24T09:37:58.515 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-24T09:37:58.531 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-24T09:37:58.531 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-24T09:37:58.633 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-24T09:37:58.633 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-24T09:37:58.812 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-fuse-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T09:37:59.086 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-fuse-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T09:37:59.086 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:59.086 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-24T09:37:59.086 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:37:59.086 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:37:59.087 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-24T09:37:59.171 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-24T09:37:59.309 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install rbd-mirror 2024-03-24T09:37:59.348 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-24T09:37:59.406 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-24T09:37:59.470 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-24T09:37:59.506 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-24T09:37:59.548 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-24T09:37:59.644 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-24T09:37:59.761 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-24T09:37:59.816 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-24T09:37:59.883 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:34 ago on Sun 24 Mar 2024 09:36:25 AM UTC. 2024-03-24T09:38:00.038 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-24T09:38:00.039 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-24T09:38:00.039 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-03-24T09:38:00.039 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-24T09:38:00.040 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-24T09:38:00.040 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 3.0 M 2024-03-24T09:38:00.040 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:38:00.040 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-24T09:38:00.040 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-24T09:38:00.040 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-24T09:38:00.040 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:38:00.040 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 3.0 M 2024-03-24T09:38:00.040 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 12 M 2024-03-24T09:38:00.040 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-24T09:38:00.575 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-24T09:38:00.629 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-24T09:38:00.663 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-24T09:38:00.696 INFO:teuthology.orchestra.run.smithi155.stdout:rbd-mirror-18.2.2-952.g738c2fd6.el9.x86_64.rpm 4.6 MB/s | 3.0 MB 00:00 2024-03-24T09:38:00.696 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:38:00.697 INFO:teuthology.orchestra.run.smithi155.stdout:Total 4.5 MB/s | 3.0 MB 00:00 2024-03-24T09:38:00.697 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-24T09:38:00.706 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-24T09:38:00.706 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-24T09:38:00.712 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-24T09:38:00.749 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-rook-2:18.2.2-952.g738c2fd6.el9.noarch 13/13 2024-03-24T09:38:00.778 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-24T09:38:00.778 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-24T09:38:01.191 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-24T09:38:01.263 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-mirror-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T09:38:01.288 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-mirror-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T09:38:01.288 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:38:01.288 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T09:38:01.288 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T09:38:01.288 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T09:38:01.288 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:38:01.494 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-952.g738c2fd6.el9.noarch 13/13 2024-03-24T09:38:01.494 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-rook-2:18.2.2-952.g738c2fd6.el9.noarch 1/13 2024-03-24T09:38:01.494 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-24T09:38:01.494 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-24T09:38:01.494 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-24T09:38:01.494 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-24T09:38:01.494 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-24T09:38:01.494 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-24T09:38:01.494 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-24T09:38:01.495 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-24T09:38:01.495 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-24T09:38:01.495 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-24T09:38:01.495 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-24T09:38:01.852 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-24T09:38:01.852 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:38:01.852 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-24T09:38:01.852 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-rook-2:18.2.2-952.g738c2fd6.el9.noarch 2024-03-24T09:38:01.852 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T09:38:01.852 INFO:teuthology.orchestra.run.smithi022.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T09:38:01.852 INFO:teuthology.orchestra.run.smithi022.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T09:38:01.852 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T09:38:01.852 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T09:38:01.853 INFO:teuthology.orchestra.run.smithi022.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T09:38:01.853 INFO:teuthology.orchestra.run.smithi022.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T09:38:01.853 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T09:38:01.853 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T09:38:01.853 INFO:teuthology.orchestra.run.smithi022.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T09:38:01.853 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T09:38:01.853 INFO:teuthology.orchestra.run.smithi022.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T09:38:01.853 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:38:01.853 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-24T09:38:02.114 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-mgr-cephadm 2024-03-24T09:38:02.352 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-mirror-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T09:38:02.352 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:38:02.352 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-24T09:38:02.353 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:38:02.353 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:38:02.353 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-24T09:38:02.605 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install rbd-nbd 2024-03-24T09:38:02.681 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:17 ago on Sun 24 Mar 2024 09:36:45 AM UTC. 2024-03-24T09:38:02.836 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-24T09:38:02.837 INFO:teuthology.orchestra.run.smithi022.stdout:======================================================================================= 2024-03-24T09:38:02.837 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-24T09:38:02.837 INFO:teuthology.orchestra.run.smithi022.stdout:======================================================================================= 2024-03-24T09:38:02.837 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-24T09:38:02.837 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-cephadm noarch 2:18.2.2-952.g738c2fd6.el9 ceph-noarch 138 k 2024-03-24T09:38:02.837 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-24T09:38:02.837 INFO:teuthology.orchestra.run.smithi022.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T09:38:02.837 INFO:teuthology.orchestra.run.smithi022.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T09:38:02.837 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T09:38:02.838 INFO:teuthology.orchestra.run.smithi022.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T09:38:02.838 INFO:teuthology.orchestra.run.smithi022.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T09:38:02.838 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:38:02.838 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-24T09:38:02.838 INFO:teuthology.orchestra.run.smithi022.stdout:======================================================================================= 2024-03-24T09:38:02.838 INFO:teuthology.orchestra.run.smithi022.stdout:Install 6 Packages 2024-03-24T09:38:02.838 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:38:02.839 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 6.8 M 2024-03-24T09:38:02.839 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 33 M 2024-03-24T09:38:02.839 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-24T09:38:03.179 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:38 ago on Sun 24 Mar 2024 09:36:25 AM UTC. 2024-03-24T09:38:03.330 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-24T09:38:03.331 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-24T09:38:03.331 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2024-03-24T09:38:03.331 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-24T09:38:03.331 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-24T09:38:03.331 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 172 k 2024-03-24T09:38:03.331 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:38:03.331 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-24T09:38:03.331 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-24T09:38:03.331 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-03-24T09:38:03.331 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:38:03.331 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 172 k 2024-03-24T09:38:03.331 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 494 k 2024-03-24T09:38:03.332 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-24T09:38:03.440 INFO:teuthology.orchestra.run.smithi022.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-03-24T09:38:03.524 INFO:teuthology.orchestra.run.smithi022.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-24T09:38:03.558 INFO:teuthology.orchestra.run.smithi155.stdout:rbd-nbd-18.2.2-952.g738c2fd6.el9.x86_64.rpm 761 kB/s | 172 kB 00:00 2024-03-24T09:38:03.558 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:38:03.559 INFO:teuthology.orchestra.run.smithi155.stdout:Total 757 kB/s | 172 kB 00:00 2024-03-24T09:38:03.559 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-24T09:38:03.567 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-24T09:38:03.567 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-24T09:38:03.623 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-24T09:38:03.623 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-24T09:38:03.674 INFO:teuthology.orchestra.run.smithi022.stdout:(3/6): ceph-mgr-cephadm-18.2.2-952.g738c2fd6.el 330 kB/s | 138 kB 00:00 2024-03-24T09:38:03.779 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-24T09:38:03.900 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-nbd-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T09:38:03.949 INFO:teuthology.orchestra.run.smithi022.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 136 kB/s | 58 kB 00:00 2024-03-24T09:38:04.100 INFO:teuthology.orchestra.run.smithi022.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 120 kB/s | 51 kB 00:00 2024-03-24T09:38:04.225 INFO:teuthology.orchestra.run.smithi022.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 701 kB/s | 548 kB 00:00 2024-03-24T09:38:04.228 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:38:04.228 INFO:teuthology.orchestra.run.smithi022.stdout:Total 4.9 MB/s | 6.8 MB 00:01 2024-03-24T09:38:04.241 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-24T09:38:04.252 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-24T09:38:04.252 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-24T09:38:04.321 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-nbd-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T09:38:04.343 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-24T09:38:04.343 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-24T09:38:04.571 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-nbd-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T09:38:04.571 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:38:04.571 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-24T09:38:04.572 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:38:04.572 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:38:04.572 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-24T09:38:04.650 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-24T09:38:04.805 DEBUG:teuthology.parallel:result is None 2024-03-24T09:38:04.823 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-24T09:38:04.895 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-24T09:38:05.446 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-24T09:38:05.532 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-24T09:38:05.601 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-24T09:38:05.631 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-952.g738c2fd6.el9.noarch 6/6 2024-03-24T09:38:06.413 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-952.g738c2fd6.el9.noarch 6/6 2024-03-24T09:38:06.413 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-952.g738c2fd6.el9.noarch 1/6 2024-03-24T09:38:06.413 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-24T09:38:06.413 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-24T09:38:06.413 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-24T09:38:06.413 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-24T09:38:06.722 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-24T09:38:06.722 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:38:06.722 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-24T09:38:06.722 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-cephadm-2:18.2.2-952.g738c2fd6.el9.noarch 2024-03-24T09:38:06.722 INFO:teuthology.orchestra.run.smithi022.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T09:38:06.722 INFO:teuthology.orchestra.run.smithi022.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T09:38:06.723 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T09:38:06.723 INFO:teuthology.orchestra.run.smithi022.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T09:38:06.723 INFO:teuthology.orchestra.run.smithi022.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T09:38:06.723 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:38:06.723 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-24T09:38:07.026 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-fuse 2024-03-24T09:38:07.597 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:22 ago on Sun 24 Mar 2024 09:36:45 AM UTC. 2024-03-24T09:38:07.753 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-24T09:38:07.754 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T09:38:07.754 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-24T09:38:07.754 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T09:38:07.754 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-24T09:38:07.754 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-fuse x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 844 k 2024-03-24T09:38:07.754 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-24T09:38:07.755 INFO:teuthology.orchestra.run.smithi022.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T09:38:07.755 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:38:07.755 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-24T09:38:07.755 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T09:38:07.755 INFO:teuthology.orchestra.run.smithi022.stdout:Install 2 Packages 2024-03-24T09:38:07.755 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:38:07.755 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 922 k 2024-03-24T09:38:07.756 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 2.7 M 2024-03-24T09:38:07.756 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-24T09:38:07.877 INFO:teuthology.orchestra.run.smithi022.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 649 kB/s | 79 kB 00:00 2024-03-24T09:38:08.060 INFO:teuthology.orchestra.run.smithi022.stdout:(2/2): ceph-fuse-18.2.2-952.g738c2fd6.el9.x86_6 2.7 MB/s | 844 kB 00:00 2024-03-24T09:38:08.062 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:38:08.062 INFO:teuthology.orchestra.run.smithi022.stdout:Total 2.9 MB/s | 922 kB 00:00 2024-03-24T09:38:08.062 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-24T09:38:08.084 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-24T09:38:08.084 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-24T09:38:08.159 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-24T09:38:08.159 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-24T09:38:08.345 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-24T09:38:08.494 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-24T09:38:08.572 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-fuse-2:18.2.2-952.g738c2fd6.el9.x86_64 2/2 2024-03-24T09:38:09.206 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-fuse-2:18.2.2-952.g738c2fd6.el9.x86_64 2/2 2024-03-24T09:38:09.206 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-fuse-2:18.2.2-952.g738c2fd6.el9.x86_64 1/2 2024-03-24T09:38:09.487 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-24T09:38:09.487 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:38:09.487 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-24T09:38:09.487 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-fuse-2:18.2.2-952.g738c2fd6.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-24T09:38:09.487 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:38:09.487 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-24T09:38:09.784 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install librados-devel 2024-03-24T09:38:10.360 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:25 ago on Sun 24 Mar 2024 09:36:45 AM UTC. 2024-03-24T09:38:10.516 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-24T09:38:10.516 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T09:38:10.517 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2024-03-24T09:38:10.517 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T09:38:10.517 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-24T09:38:10.517 INFO:teuthology.orchestra.run.smithi022.stdout: librados-devel x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 127 k 2024-03-24T09:38:10.517 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:38:10.517 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-24T09:38:10.517 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T09:38:10.517 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-24T09:38:10.517 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:38:10.517 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 127 k 2024-03-24T09:38:10.517 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 456 k 2024-03-24T09:38:10.517 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-24T09:38:10.874 INFO:teuthology.orchestra.run.smithi022.stdout:librados-devel-18.2.2-952.g738c2fd6.el9.x86_64. 355 kB/s | 127 kB 00:00 2024-03-24T09:38:10.875 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:38:10.875 INFO:teuthology.orchestra.run.smithi022.stdout:Total 354 kB/s | 127 kB 00:00 2024-03-24T09:38:10.875 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-24T09:38:10.881 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-24T09:38:10.881 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-24T09:38:10.938 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-24T09:38:10.939 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-24T09:38:11.087 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-24T09:38:11.200 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : librados-devel-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T09:38:11.644 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librados-devel-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T09:38:11.937 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librados-devel-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T09:38:11.937 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:38:11.937 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-24T09:38:11.937 INFO:teuthology.orchestra.run.smithi022.stdout: librados-devel-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:38:11.937 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:38:11.937 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-24T09:38:12.196 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install libcephfs2 2024-03-24T09:38:12.775 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:27 ago on Sun 24 Mar 2024 09:36:45 AM UTC. 2024-03-24T09:38:12.875 INFO:teuthology.orchestra.run.smithi022.stdout:Package libcephfs2-2:18.2.2-952.g738c2fd6.el9.x86_64 is already installed. 2024-03-24T09:38:12.931 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-24T09:38:12.932 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2024-03-24T09:38:12.932 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-24T09:38:12.995 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install libcephfs-devel 2024-03-24T09:38:13.567 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:28 ago on Sun 24 Mar 2024 09:36:45 AM UTC. 2024-03-24T09:38:13.721 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-24T09:38:13.722 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T09:38:13.722 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2024-03-24T09:38:13.722 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T09:38:13.722 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-24T09:38:13.722 INFO:teuthology.orchestra.run.smithi022.stdout: libcephfs-devel x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 31 k 2024-03-24T09:38:13.722 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:38:13.722 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-24T09:38:13.723 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T09:38:13.723 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-24T09:38:13.723 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:38:13.723 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 31 k 2024-03-24T09:38:13.723 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 139 k 2024-03-24T09:38:13.723 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-24T09:38:13.886 INFO:teuthology.orchestra.run.smithi022.stdout:libcephfs-devel-18.2.2-952.g738c2fd6.el9.x86_64 190 kB/s | 31 kB 00:00 2024-03-24T09:38:13.887 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:38:13.887 INFO:teuthology.orchestra.run.smithi022.stdout:Total 188 kB/s | 31 kB 00:00 2024-03-24T09:38:13.887 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-24T09:38:13.891 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-24T09:38:13.891 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-24T09:38:13.913 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-24T09:38:13.914 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-24T09:38:13.999 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-24T09:38:14.137 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libcephfs-devel-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T09:38:14.542 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T09:38:14.794 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libcephfs-devel-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T09:38:14.794 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:38:14.794 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-24T09:38:14.794 INFO:teuthology.orchestra.run.smithi022.stdout: libcephfs-devel-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:38:14.794 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:38:14.794 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-24T09:38:14.979 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install librados2 2024-03-24T09:38:15.552 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:30 ago on Sun 24 Mar 2024 09:36:45 AM UTC. 2024-03-24T09:38:15.653 INFO:teuthology.orchestra.run.smithi022.stdout:Package librados2-2:18.2.2-952.g738c2fd6.el9.x86_64 is already installed. 2024-03-24T09:38:15.708 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-24T09:38:15.709 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2024-03-24T09:38:15.709 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-24T09:38:15.775 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install librbd1 2024-03-24T09:38:16.347 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:31 ago on Sun 24 Mar 2024 09:36:45 AM UTC. 2024-03-24T09:38:16.447 INFO:teuthology.orchestra.run.smithi022.stdout:Package librbd1-2:18.2.2-952.g738c2fd6.el9.x86_64 is already installed. 2024-03-24T09:38:16.504 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-24T09:38:16.505 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2024-03-24T09:38:16.505 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-24T09:38:16.564 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install python3-rados 2024-03-24T09:38:17.136 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:32 ago on Sun 24 Mar 2024 09:36:45 AM UTC. 2024-03-24T09:38:17.235 INFO:teuthology.orchestra.run.smithi022.stdout:Package python3-rados-2:18.2.2-952.g738c2fd6.el9.x86_64 is already installed. 2024-03-24T09:38:17.291 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-24T09:38:17.292 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2024-03-24T09:38:17.292 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-24T09:38:17.366 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install python3-rgw 2024-03-24T09:38:17.939 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:32 ago on Sun 24 Mar 2024 09:36:45 AM UTC. 2024-03-24T09:38:18.038 INFO:teuthology.orchestra.run.smithi022.stdout:Package python3-rgw-2:18.2.2-952.g738c2fd6.el9.x86_64 is already installed. 2024-03-24T09:38:18.094 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-24T09:38:18.095 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2024-03-24T09:38:18.095 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-24T09:38:18.163 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install python3-cephfs 2024-03-24T09:38:18.738 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:33 ago on Sun 24 Mar 2024 09:36:45 AM UTC. 2024-03-24T09:38:18.837 INFO:teuthology.orchestra.run.smithi022.stdout:Package python3-cephfs-2:18.2.2-952.g738c2fd6.el9.x86_64 is already installed. 2024-03-24T09:38:18.892 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-24T09:38:18.893 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2024-03-24T09:38:18.893 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-24T09:38:18.958 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install python3-rbd 2024-03-24T09:38:19.528 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:34 ago on Sun 24 Mar 2024 09:36:45 AM UTC. 2024-03-24T09:38:19.629 INFO:teuthology.orchestra.run.smithi022.stdout:Package python3-rbd-2:18.2.2-952.g738c2fd6.el9.x86_64 is already installed. 2024-03-24T09:38:19.684 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-24T09:38:19.685 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2024-03-24T09:38:19.685 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-24T09:38:19.749 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install rbd-fuse 2024-03-24T09:38:20.318 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:35 ago on Sun 24 Mar 2024 09:36:45 AM UTC. 2024-03-24T09:38:20.473 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-24T09:38:20.474 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T09:38:20.474 INFO:teuthology.orchestra.run.smithi022.stdout: Package Architecture Version Repository Size 2024-03-24T09:38:20.474 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T09:38:20.474 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-24T09:38:20.474 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-fuse x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 87 k 2024-03-24T09:38:20.474 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:38:20.474 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-24T09:38:20.475 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T09:38:20.475 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-24T09:38:20.475 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:38:20.475 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 87 k 2024-03-24T09:38:20.475 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 227 k 2024-03-24T09:38:20.475 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-24T09:38:20.798 INFO:teuthology.orchestra.run.smithi022.stdout:rbd-fuse-18.2.2-952.g738c2fd6.el9.x86_64.rpm 268 kB/s | 87 kB 00:00 2024-03-24T09:38:20.798 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:38:20.798 INFO:teuthology.orchestra.run.smithi022.stdout:Total 267 kB/s | 87 kB 00:00 2024-03-24T09:38:20.798 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-24T09:38:20.806 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-24T09:38:20.806 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-24T09:38:20.863 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-24T09:38:20.863 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-24T09:38:21.003 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-24T09:38:21.107 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : rbd-fuse-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T09:38:21.473 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: rbd-fuse-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T09:38:21.758 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : rbd-fuse-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T09:38:21.758 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:38:21.758 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-24T09:38:21.759 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-fuse-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:38:21.759 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:38:21.759 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-24T09:38:21.989 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install rbd-mirror 2024-03-24T09:38:22.563 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:37 ago on Sun 24 Mar 2024 09:36:45 AM UTC. 2024-03-24T09:38:22.717 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-24T09:38:22.718 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T09:38:22.718 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2024-03-24T09:38:22.718 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T09:38:22.718 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-24T09:38:22.718 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-mirror x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 3.0 M 2024-03-24T09:38:22.718 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:38:22.718 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-24T09:38:22.718 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T09:38:22.718 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-24T09:38:22.719 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:38:22.719 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 3.0 M 2024-03-24T09:38:22.719 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 12 M 2024-03-24T09:38:22.719 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-24T09:38:23.120 INFO:teuthology.orchestra.run.smithi022.stdout:rbd-mirror-18.2.2-952.g738c2fd6.el9.x86_64.rpm 7.5 MB/s | 3.0 MB 00:00 2024-03-24T09:38:23.121 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:38:23.121 INFO:teuthology.orchestra.run.smithi022.stdout:Total 7.4 MB/s | 3.0 MB 00:00 2024-03-24T09:38:23.121 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-24T09:38:23.131 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-24T09:38:23.131 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-24T09:38:23.207 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-24T09:38:23.207 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-24T09:38:23.634 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-24T09:38:23.696 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : rbd-mirror-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T09:38:23.719 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: rbd-mirror-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T09:38:23.719 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:38:23.719 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T09:38:23.719 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T09:38:23.719 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T09:38:23.719 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:38:24.858 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : rbd-mirror-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T09:38:24.858 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:38:24.858 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-24T09:38:24.858 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-mirror-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:38:24.858 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:38:24.858 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-24T09:38:25.099 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install rbd-nbd 2024-03-24T09:38:25.676 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:40 ago on Sun 24 Mar 2024 09:36:45 AM UTC. 2024-03-24T09:38:25.834 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-24T09:38:25.835 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T09:38:25.835 INFO:teuthology.orchestra.run.smithi022.stdout: Package Architecture Version Repository Size 2024-03-24T09:38:25.835 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T09:38:25.835 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-24T09:38:25.835 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-nbd x86_64 2:18.2.2-952.g738c2fd6.el9 ceph 172 k 2024-03-24T09:38:25.835 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:38:25.835 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-24T09:38:25.835 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-24T09:38:25.836 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-24T09:38:25.836 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:38:25.836 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 172 k 2024-03-24T09:38:25.836 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 494 k 2024-03-24T09:38:25.836 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-24T09:38:26.190 INFO:teuthology.orchestra.run.smithi022.stdout:rbd-nbd-18.2.2-952.g738c2fd6.el9.x86_64.rpm 484 kB/s | 172 kB 00:00 2024-03-24T09:38:26.191 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:38:26.191 INFO:teuthology.orchestra.run.smithi022.stdout:Total 482 kB/s | 172 kB 00:00 2024-03-24T09:38:26.191 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-24T09:38:26.200 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-24T09:38:26.200 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-24T09:38:26.258 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-24T09:38:26.258 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-24T09:38:26.405 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-24T09:38:26.499 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : rbd-nbd-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T09:38:26.927 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: rbd-nbd-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T09:38:27.201 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : rbd-nbd-2:18.2.2-952.g738c2fd6.el9.x86_64 1/1 2024-03-24T09:38:27.201 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:38:27.201 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-24T09:38:27.201 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-nbd-2:18.2.2-952.g738c2fd6.el9.x86_64 2024-03-24T09:38:27.201 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:38:27.201 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-24T09:38:27.429 DEBUG:teuthology.parallel:result is None 2024-03-24T09:38:27.429 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T09:38:27.429 INFO:teuthology.packaging:ref: None 2024-03-24T09:38:27.429 INFO:teuthology.packaging:tag: None 2024-03-24T09:38:27.430 INFO:teuthology.packaging:branch: reef 2024-03-24T09:38:27.430 INFO:teuthology.packaging:sha1: 21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T09:38:27.430 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-03-24T09:38:27.773 DEBUG:teuthology.orchestra.run.smithi022:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T09:38:27.795 INFO:teuthology.orchestra.run.smithi022.stdout:18.2.2-952.g738c2fd6.el9 2024-03-24T09:38:27.795 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-952.g738c2fd6.el9 2024-03-24T09:38:27.795 INFO:teuthology.task.install:The correct ceph version 18.2.2-952.g738c2fd6 is installed. 2024-03-24T09:38:27.797 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T09:38:27.797 INFO:teuthology.packaging:ref: None 2024-03-24T09:38:27.797 INFO:teuthology.packaging:tag: None 2024-03-24T09:38:27.797 INFO:teuthology.packaging:branch: reef 2024-03-24T09:38:27.797 INFO:teuthology.packaging:sha1: 21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T09:38:27.798 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-03-24T09:38:28.136 DEBUG:teuthology.orchestra.run.smithi155:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T09:38:28.161 INFO:teuthology.orchestra.run.smithi155.stdout:18.2.2-952.g738c2fd6.el9 2024-03-24T09:38:28.161 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-952.g738c2fd6.el9 2024-03-24T09:38:28.161 INFO:teuthology.task.install:The correct ceph version 18.2.2-952.g738c2fd6 is installed. 2024-03-24T09:38:28.163 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-24T09:38:28.163 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T09:38:28.163 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T09:38:28.195 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-24T09:38:28.195 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T09:38:28.234 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-24T09:38:28.234 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T09:38:28.234 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T09:38:28.264 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T09:38:28.333 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-24T09:38:28.333 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T09:38:28.361 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T09:38:28.426 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-24T09:38:28.427 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T09:38:28.427 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T09:38:28.455 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T09:38:28.523 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-24T09:38:28.523 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T09:38:28.551 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T09:38:28.619 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-24T09:38:28.619 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T09:38:28.619 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T09:38:28.647 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T09:38:28.715 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-24T09:38:28.715 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T09:38:28.742 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T09:38:28.809 INFO:teuthology.run_tasks:Running task print... 2024-03-24T09:38:28.818 INFO:teuthology.task.print:**** done install task... 2024-03-24T09:38:28.818 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-24T09:38:28.911 INFO:tasks.cephadm:Config: {'compiled_cephadm_branch': 'reef', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:reef', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de'} 2024-03-24T09:38:28.911 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:reef 2024-03-24T09:38:28.911 INFO:tasks.cephadm:Cluster fsid is 45356448-e9c2-11ee-95cd-87774f69a715 2024-03-24T09:38:28.912 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-24T09:38:28.912 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-03-24T09:38:28.912 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi022': '172.21.15.22', 'mon.smithi155': '172.21.15.155'} 2024-03-24T09:38:28.912 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-24T09:38:28.912 DEBUG:teuthology.orchestra.run.smithi022:> sudo hostname $(hostname -s) 2024-03-24T09:38:28.940 DEBUG:teuthology.orchestra.run.smithi155:> sudo hostname $(hostname -s) 2024-03-24T09:38:28.968 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra for reef 2024-03-24T09:38:28.968 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T09:38:29.108 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77900/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1855-g21b3bd67', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1855.g21b3bd67'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-23 04:48:23.862772', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-batrick-testing-20240323.003144-squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-24T09:38:29.424 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref reef, sha1 738c2fd67dbc0b17e2e99568a03974712d998229 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&ref=reef 2024-03-24T09:38:29.425 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/reef/738c2fd67dbc0b17e2e99568a03974712d998229/centos/9/x86_64/flavors/default/cephadm 2024-03-24T09:38:29.425 DEBUG:teuthology.orchestra.run.smithi022:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/reef/738c2fd67dbc0b17e2e99568a03974712d998229/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T09:38:29.992 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r--. 1 ubuntu ubuntu 215105 Mar 24 09:38 /home/ubuntu/cephtest/cephadm 2024-03-24T09:38:29.992 DEBUG:teuthology.orchestra.run.smithi155:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/reef/738c2fd67dbc0b17e2e99568a03974712d998229/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T09:38:30.405 INFO:teuthology.orchestra.run.smithi155.stdout:-rw-r--r--. 1 ubuntu ubuntu 215105 Mar 24 09:38 /home/ubuntu/cephtest/cephadm 2024-03-24T09:38:30.405 DEBUG:teuthology.orchestra.run.smithi022:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T09:38:30.424 DEBUG:teuthology.orchestra.run.smithi155:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T09:38:30.452 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:reef on all hosts... 2024-03-24T09:38:30.452 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull 2024-03-24T09:38:30.466 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull 2024-03-24T09:38:31.308 INFO:teuthology.orchestra.run.smithi155.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-03-24T09:38:31.368 INFO:teuthology.orchestra.run.smithi022.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-03-24T09:39:15.190 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T09:39:15.191 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph_version": "ceph version 18.2.2-952-g738c2fd6 (738c2fd67dbc0b17e2e99568a03974712d998229) reef (stable)", 2024-03-24T09:39:15.191 INFO:teuthology.orchestra.run.smithi022.stdout: "image_id": "68aff9aff3b2fe3c16548acc72564779ed9f377b279f3076ce38885b01df0d46", 2024-03-24T09:39:15.191 INFO:teuthology.orchestra.run.smithi022.stdout: "repo_digests": [ 2024-03-24T09:39:15.191 INFO:teuthology.orchestra.run.smithi022.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d" 2024-03-24T09:39:15.191 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2024-03-24T09:39:15.191 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T09:39:16.487 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-03-24T09:39:16.488 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_version": "ceph version 18.2.2-952-g738c2fd6 (738c2fd67dbc0b17e2e99568a03974712d998229) reef (stable)", 2024-03-24T09:39:16.488 INFO:teuthology.orchestra.run.smithi155.stdout: "image_id": "68aff9aff3b2fe3c16548acc72564779ed9f377b279f3076ce38885b01df0d46", 2024-03-24T09:39:16.488 INFO:teuthology.orchestra.run.smithi155.stdout: "repo_digests": [ 2024-03-24T09:39:16.488 INFO:teuthology.orchestra.run.smithi155.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d" 2024-03-24T09:39:16.488 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-03-24T09:39:16.488 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-03-24T09:39:16.506 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /etc/ceph 2024-03-24T09:39:16.542 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /etc/ceph 2024-03-24T09:39:16.571 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 777 /etc/ceph 2024-03-24T09:39:16.612 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 777 /etc/ceph 2024-03-24T09:39:16.639 INFO:tasks.cephadm:Writing seed config... 2024-03-24T09:39:16.640 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-03-24T09:39:16.640 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-03-24T09:39:16.640 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-03-24T09:39:16.640 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-03-24T09:39:16.640 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-03-24T09:39:16.640 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-24T09:39:16.640 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-24T09:39:16.640 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-03-24T09:39:16.641 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-03-24T09:39:16.641 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-24T09:39:16.641 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-24T09:39:16.641 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-03-24T09:39:16.641 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-24T09:39:16.641 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-24T09:39:16.641 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-24T09:39:16.641 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-24T09:39:16.641 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-24T09:39:16.641 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-24T09:39:16.641 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-24T09:39:16.641 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-24T09:39:16.641 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-24T09:39:16.641 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-24T09:39:16.641 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-24T09:39:16.641 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-03-24T09:39:16.642 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-24T09:39:16.642 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-24T09:39:16.642 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-24T09:39:16.642 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-24T09:39:16.642 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-24T09:39:16.642 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-24T09:39:16.642 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-24T09:39:16.642 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-24T09:39:16.642 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-24T09:39:16.642 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-24T09:39:16.642 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-03-24T09:39:16.642 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-24T09:39:16.642 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-24T09:39:16.643 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-24T09:39:16.643 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-24T09:39:16.643 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-24T09:39:16.643 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-03-24T09:39:16.643 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-24T09:39:16.643 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T09:39:16.644 DEBUG:teuthology.orchestra.run.smithi022:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-24T09:39:16.669 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 45356448-e9c2-11ee-95cd-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-24T09:39:16.670 DEBUG:teuthology.orchestra.run.smithi022:mon.smithi022> sudo journalctl -f -n 0 -u ceph-45356448-e9c2-11ee-95cd-87774f69a715@mon.smithi022.service 2024-03-24T09:39:16.712 INFO:tasks.cephadm:Bootstrapping... 2024-03-24T09:39:16.712 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef -v bootstrap --fsid 45356448-e9c2-11ee-95cd-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.22 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-24T09:39:16.877 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:39:16.877 INFO:teuthology.orchestra.run.smithi022.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:reef', '-v', 'bootstrap', '--fsid', '45356448-e9c2-11ee-95cd-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.22', '--skip-admin-label'] 2024-03-24T09:39:16.905 INFO:teuthology.orchestra.run.smithi022.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T09:39:16.905 INFO:teuthology.orchestra.run.smithi022.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-24T09:39:16.906 INFO:teuthology.orchestra.run.smithi022.stdout:Verifying podman|docker is present... 2024-03-24T09:39:16.930 INFO:teuthology.orchestra.run.smithi022.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T09:39:16.930 INFO:teuthology.orchestra.run.smithi022.stdout:Verifying lvm2 is present... 2024-03-24T09:39:16.930 INFO:teuthology.orchestra.run.smithi022.stdout:Verifying time synchronization is in place... 2024-03-24T09:39:16.937 INFO:teuthology.orchestra.run.smithi022.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-24T09:39:16.937 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-24T09:39:16.943 INFO:teuthology.orchestra.run.smithi022.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-24T09:39:16.943 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stdout inactive 2024-03-24T09:39:16.949 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stdout enabled 2024-03-24T09:39:16.955 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stdout active 2024-03-24T09:39:16.955 INFO:teuthology.orchestra.run.smithi022.stdout:Unit chronyd.service is enabled and running 2024-03-24T09:39:16.955 INFO:teuthology.orchestra.run.smithi022.stdout:Repeating the final host check... 2024-03-24T09:39:16.978 INFO:teuthology.orchestra.run.smithi022.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T09:39:16.978 INFO:teuthology.orchestra.run.smithi022.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-24T09:39:16.978 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl is present 2024-03-24T09:39:16.978 INFO:teuthology.orchestra.run.smithi022.stdout:lvcreate is present 2024-03-24T09:39:16.984 INFO:teuthology.orchestra.run.smithi022.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-24T09:39:16.984 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-24T09:39:16.990 INFO:teuthology.orchestra.run.smithi022.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-24T09:39:16.990 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stdout inactive 2024-03-24T09:39:16.996 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stdout enabled 2024-03-24T09:39:17.002 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stdout active 2024-03-24T09:39:17.002 INFO:teuthology.orchestra.run.smithi022.stdout:Unit chronyd.service is enabled and running 2024-03-24T09:39:17.002 INFO:teuthology.orchestra.run.smithi022.stdout:Host looks OK 2024-03-24T09:39:17.002 INFO:teuthology.orchestra.run.smithi022.stdout:Cluster fsid: 45356448-e9c2-11ee-95cd-87774f69a715 2024-03-24T09:39:17.002 INFO:teuthology.orchestra.run.smithi022.stdout:Acquiring lock 139763844960608 on /run/cephadm/45356448-e9c2-11ee-95cd-87774f69a715.lock 2024-03-24T09:39:17.002 INFO:teuthology.orchestra.run.smithi022.stdout:Lock 139763844960608 acquired on /run/cephadm/45356448-e9c2-11ee-95cd-87774f69a715.lock 2024-03-24T09:39:17.003 INFO:teuthology.orchestra.run.smithi022.stdout:Verifying IP 172.21.15.22 port 3300 ... 2024-03-24T09:39:17.003 INFO:teuthology.orchestra.run.smithi022.stdout:Verifying IP 172.21.15.22 port 6789 ... 2024-03-24T09:39:17.003 INFO:teuthology.orchestra.run.smithi022.stdout:Base mon IP(s) is [172.21.15.22:3300, 172.21.15.22:6789], mon addrv is [v2:172.21.15.22:3300,v1:172.21.15.22:6789] 2024-03-24T09:39:17.006 INFO:teuthology.orchestra.run.smithi022.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.22 metric 100 2024-03-24T09:39:17.006 INFO:teuthology.orchestra.run.smithi022.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.22 metric 100 2024-03-24T09:39:17.009 INFO:teuthology.orchestra.run.smithi022.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-24T09:39:17.009 INFO:teuthology.orchestra.run.smithi022.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-24T09:39:17.009 INFO:teuthology.orchestra.run.smithi022.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-24T09:39:17.011 INFO:teuthology.orchestra.run.smithi022.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-24T09:39:17.011 INFO:teuthology.orchestra.run.smithi022.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-24T09:39:17.011 INFO:teuthology.orchestra.run.smithi022.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-24T09:39:17.011 INFO:teuthology.orchestra.run.smithi022.stdout:/sbin/ip: stdout 3: ens1f0: mtu 1500 state UP qlen 1000 2024-03-24T09:39:17.011 INFO:teuthology.orchestra.run.smithi022.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe59:5de4/64 scope link noprefixroute 2024-03-24T09:39:17.011 INFO:teuthology.orchestra.run.smithi022.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-24T09:39:17.012 INFO:teuthology.orchestra.run.smithi022.stdout:Mon IP `172.21.15.22` is in CIDR network `172.21.0.0/20` 2024-03-24T09:39:17.012 INFO:teuthology.orchestra.run.smithi022.stdout:Mon IP `172.21.15.22` is in CIDR network `172.21.0.0/20` 2024-03-24T09:39:17.012 INFO:teuthology.orchestra.run.smithi022.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-24T09:39:17.013 INFO:teuthology.orchestra.run.smithi022.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-24T09:39:17.013 INFO:teuthology.orchestra.run.smithi022.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-03-24T09:39:17.507 INFO:teuthology.orchestra.run.smithi022.stdout:/bin/podman: stdout 68aff9aff3b2fe3c16548acc72564779ed9f377b279f3076ce38885b01df0d46 2024-03-24T09:39:17.508 INFO:teuthology.orchestra.run.smithi022.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:reef... 2024-03-24T09:39:17.508 INFO:teuthology.orchestra.run.smithi022.stdout:/bin/podman: stderr Getting image source signatures 2024-03-24T09:39:17.508 INFO:teuthology.orchestra.run.smithi022.stdout:/bin/podman: stderr Copying blob sha256:8566210fcee75c6a55e6eeb2f06888d926f27e0de261f0bce516c1cbd24b1704 2024-03-24T09:39:17.508 INFO:teuthology.orchestra.run.smithi022.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-24T09:39:17.508 INFO:teuthology.orchestra.run.smithi022.stdout:/bin/podman: stderr Copying config sha256:68aff9aff3b2fe3c16548acc72564779ed9f377b279f3076ce38885b01df0d46 2024-03-24T09:39:17.508 INFO:teuthology.orchestra.run.smithi022.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-24T09:39:18.710 INFO:teuthology.orchestra.run.smithi022.stdout:ceph: stdout ceph version 18.2.2-952-g738c2fd6 (738c2fd67dbc0b17e2e99568a03974712d998229) reef (stable) 2024-03-24T09:39:18.711 INFO:teuthology.orchestra.run.smithi022.stdout:Ceph version: ceph version 18.2.2-952-g738c2fd6 (738c2fd67dbc0b17e2e99568a03974712d998229) reef (stable) 2024-03-24T09:39:18.711 INFO:teuthology.orchestra.run.smithi022.stdout:Extracting ceph user uid/gid from container image... 2024-03-24T09:39:19.552 INFO:teuthology.orchestra.run.smithi022.stdout:stat: stdout 167 167 2024-03-24T09:39:19.552 INFO:teuthology.orchestra.run.smithi022.stdout:Creating initial keys... 2024-03-24T09:39:20.390 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph-authtool: stdout AQDH9P9lfSTkNRAAPCo7r1Al8Niu1CFCFMCz8w== 2024-03-24T09:39:21.281 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph-authtool: stdout AQDI9P9la/nDKhAAeuiaJi/EW//NQ3K5SrGrig== 2024-03-24T09:39:22.235 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph-authtool: stdout AQDJ9P9lzLkSJBAAcQmZNhjfkSnQutAKJfrwnA== 2024-03-24T09:39:22.236 INFO:teuthology.orchestra.run.smithi022.stdout:Creating initial monmap... 2024-03-24T09:39:23.142 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-24T09:39:23.142 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-03-24T09:39:23.142 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 45356448-e9c2-11ee-95cd-87774f69a715 2024-03-24T09:39:23.142 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-24T09:39:23.142 INFO:teuthology.orchestra.run.smithi022.stdout:monmaptool for smithi022 [v2:172.21.15.22:3300,v1:172.21.15.22:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-24T09:39:23.142 INFO:teuthology.orchestra.run.smithi022.stdout:setting min_mon_release = pacific 2024-03-24T09:39:23.142 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/monmaptool: set fsid to 45356448-e9c2-11ee-95cd-87774f69a715 2024-03-24T09:39:23.142 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-24T09:39:23.142 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:39:23.143 INFO:teuthology.orchestra.run.smithi022.stdout:Creating mon... 2024-03-24T09:39:24.080 INFO:teuthology.orchestra.run.smithi022.stdout:create mon.smithi022 on 2024-03-24T09:39:24.298 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-24T09:39:24.441 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-24T09:39:24.635 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-45356448-e9c2-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-45356448-e9c2-11ee-95cd-87774f69a715.target. 2024-03-24T09:39:24.636 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-45356448-e9c2-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-45356448-e9c2-11ee-95cd-87774f69a715.target. 2024-03-24T09:39:24.840 INFO:teuthology.orchestra.run.smithi022.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-45356448-e9c2-11ee-95cd-87774f69a715@mon.smithi022 2024-03-24T09:39:24.840 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stderr Failed to reset failed state of unit ceph-45356448-e9c2-11ee-95cd-87774f69a715@mon.smithi022.service: Unit ceph-45356448-e9c2-11ee-95cd-87774f69a715@mon.smithi022.service not loaded. 2024-03-24T09:39:25.002 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-45356448-e9c2-11ee-95cd-87774f69a715.target.wants/ceph-45356448-e9c2-11ee-95cd-87774f69a715@mon.smithi022.service → /etc/systemd/system/ceph-45356448-e9c2-11ee-95cd-87774f69a715@.service. 2024-03-24T09:39:25.519 INFO:teuthology.orchestra.run.smithi022.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T09:39:25.519 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stdout disabled 2024-03-24T09:39:25.527 INFO:teuthology.orchestra.run.smithi022.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T09:39:25.527 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stdout inactive 2024-03-24T09:39:25.527 INFO:teuthology.orchestra.run.smithi022.stdout:firewalld.service is not enabled 2024-03-24T09:39:25.527 INFO:teuthology.orchestra.run.smithi022.stdout:Not possible to enable service . firewalld.service is not available 2024-03-24T09:39:25.527 INFO:teuthology.orchestra.run.smithi022.stdout:Waiting for mon to start... 2024-03-24T09:39:25.528 INFO:teuthology.orchestra.run.smithi022.stdout:Waiting for mon... 2024-03-24T09:39:26.606 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout cluster: 2024-03-24T09:39:26.606 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout id: 45356448-e9c2-11ee-95cd-87774f69a715 2024-03-24T09:39:26.606 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-24T09:39:26.606 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout 2024-03-24T09:39:26.606 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout services: 2024-03-24T09:39:26.606 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi022 (age 0.557347s) 2024-03-24T09:39:26.606 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-24T09:39:26.606 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-24T09:39:26.606 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout 2024-03-24T09:39:26.606 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout data: 2024-03-24T09:39:26.606 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-24T09:39:26.607 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-24T09:39:26.607 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-24T09:39:26.607 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout pgs: 2024-03-24T09:39:26.607 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout 2024-03-24T09:39:26.607 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:25.977+0000 7fe6ced60700 1 Processor -- start 2024-03-24T09:39:26.607 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:25.977+0000 7fe6ced60700 1 -- start start 2024-03-24T09:39:26.607 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:25.977+0000 7fe6ced60700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe6c8106f30 0x7fe6c8109320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:39:26.607 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:25.977+0000 7fe6ced60700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c80745f0 con 0x7fe6c8106f30 2024-03-24T09:39:26.607 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:25.977+0000 7fe6ccafc700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe6c8106f30 0x7fe6c8109320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:39:26.607 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:25.978+0000 7fe6ccafc700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe6c8106f30 0x7fe6c8109320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:53358/0 (socket says 172.21.15.22:53358) 2024-03-24T09:39:26.607 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:25.978+0000 7fe6ccafc700 1 -- 172.21.15.22:0/845474492 learned_addr learned my addr 172.21.15.22:0/845474492 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:39:26.607 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:25.978+0000 7fe6ccafc700 1 -- 172.21.15.22:0/845474492 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c8109860 con 0x7fe6c8106f30 2024-03-24T09:39:26.607 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:25.978+0000 7fe6ccafc700 1 --2- 172.21.15.22:0/845474492 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe6c8106f30 0x7fe6c8109320 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc0098a0 tx=0x7fe6bc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e83e5f390ce1e8d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:39:26.608 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:25.978+0000 7fe6c77fe700 1 -- 172.21.15.22:0/845474492 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6bc004180 con 0x7fe6c8106f30 2024-03-24T09:39:26.608 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:25.978+0000 7fe6c77fe700 1 -- 172.21.15.22:0/845474492 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6bc0042e0 con 0x7fe6c8106f30 2024-03-24T09:39:26.608 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:25.978+0000 7fe6c77fe700 1 -- 172.21.15.22:0/845474492 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6bc0036a0 con 0x7fe6c8106f30 2024-03-24T09:39:26.608 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:25.979+0000 7fe6ced60700 1 -- 172.21.15.22:0/845474492 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe6c8106f30 msgr2=0x7fe6c8109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:39:26.608 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:25.979+0000 7fe6ced60700 1 --2- 172.21.15.22:0/845474492 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe6c8106f30 0x7fe6c8109320 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc0098a0 tx=0x7fe6bc009bb0 comp rx=0 tx=0).stop 2024-03-24T09:39:26.608 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:25.979+0000 7fe6ced60700 1 -- 172.21.15.22:0/845474492 shutdown_connections 2024-03-24T09:39:26.608 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:25.979+0000 7fe6ced60700 1 --2- 172.21.15.22:0/845474492 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe6c8106f30 0x7fe6c8109320 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:39:26.608 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:25.979+0000 7fe6ced60700 1 -- 172.21.15.22:0/845474492 >> 172.21.15.22:0/845474492 conn(0x7fe6c81009c0 msgr2=0x7fe6c8102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:39:26.608 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:25.979+0000 7fe6ced60700 1 -- 172.21.15.22:0/845474492 shutdown_connections 2024-03-24T09:39:26.608 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:25.979+0000 7fe6ced60700 1 -- 172.21.15.22:0/845474492 wait complete. 2024-03-24T09:39:26.608 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:25.980+0000 7fe6ced60700 1 Processor -- start 2024-03-24T09:39:26.608 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:25.980+0000 7fe6ced60700 1 -- start start 2024-03-24T09:39:26.608 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:25.980+0000 7fe6ced60700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe6c8106f30 0x7fe6c8199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:39:26.608 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:25.980+0000 7fe6ced60700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c8199ec0 con 0x7fe6c8106f30 2024-03-24T09:39:26.609 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:25.980+0000 7fe6ccafc700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe6c8106f30 0x7fe6c8199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:39:26.609 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:25.980+0000 7fe6ccafc700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe6c8106f30 0x7fe6c8199980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:53372/0 (socket says 172.21.15.22:53372) 2024-03-24T09:39:26.609 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:25.980+0000 7fe6ccafc700 1 -- 172.21.15.22:0/1387499479 learned_addr learned my addr 172.21.15.22:0/1387499479 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:39:26.609 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:25.980+0000 7fe6ccafc700 1 -- 172.21.15.22:0/1387499479 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6bc009550 con 0x7fe6c8106f30 2024-03-24T09:39:26.609 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:25.981+0000 7fe6ccafc700 1 --2- 172.21.15.22:0/1387499479 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe6c8106f30 0x7fe6c8199980 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8073a30 tx=0x7fe6bc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:39:26.609 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:25.981+0000 7fe6c5ffb700 1 -- 172.21.15.22:0/1387499479 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6bc003ce0 con 0x7fe6c8106f30 2024-03-24T09:39:26.609 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:25.981+0000 7fe6c5ffb700 1 -- 172.21.15.22:0/1387499479 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6bc020070 con 0x7fe6c8106f30 2024-03-24T09:39:26.609 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:25.981+0000 7fe6c5ffb700 1 -- 172.21.15.22:0/1387499479 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6bc011630 con 0x7fe6c8106f30 2024-03-24T09:39:26.609 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:25.981+0000 7fe6ced60700 1 -- 172.21.15.22:0/1387499479 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6c819a0c0 con 0x7fe6c8106f30 2024-03-24T09:39:26.609 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:25.981+0000 7fe6ced60700 1 -- 172.21.15.22:0/1387499479 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6c819a560 con 0x7fe6c8106f30 2024-03-24T09:39:26.609 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:25.981+0000 7fe6c5ffb700 1 -- 172.21.15.22:0/1387499479 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fe6bc022030 con 0x7fe6c8106f30 2024-03-24T09:39:26.609 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:25.981+0000 7fe6c5ffb700 1 -- 172.21.15.22:0/1387499479 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe6bc025e20 con 0x7fe6c8106f30 2024-03-24T09:39:26.609 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:25.982+0000 7fe6ced60700 1 -- 172.21.15.22:0/1387499479 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6c81937c0 con 0x7fe6c8106f30 2024-03-24T09:39:26.610 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:25.984+0000 7fe6c5ffb700 1 -- 172.21.15.22:0/1387499479 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fe6bc025480 con 0x7fe6c8106f30 2024-03-24T09:39:26.610 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:26.057+0000 7fe6ced60700 1 -- 172.21.15.22:0/1387499479 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fe6c802cf20 con 0x7fe6c8106f30 2024-03-24T09:39:26.610 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:26.057+0000 7fe6c5ffb700 1 -- 172.21.15.22:0/1387499479 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7fe6bc011c20 con 0x7fe6c8106f30 2024-03-24T09:39:26.610 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:26.058+0000 7fe6ced60700 1 -- 172.21.15.22:0/1387499479 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe6c8106f30 msgr2=0x7fe6c8199980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:39:26.610 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:26.058+0000 7fe6ced60700 1 --2- 172.21.15.22:0/1387499479 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe6c8106f30 0x7fe6c8199980 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8073a30 tx=0x7fe6bc004750 comp rx=0 tx=0).stop 2024-03-24T09:39:26.610 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:26.058+0000 7fe6ced60700 1 -- 172.21.15.22:0/1387499479 shutdown_connections 2024-03-24T09:39:26.610 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:26.058+0000 7fe6ced60700 1 --2- 172.21.15.22:0/1387499479 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe6c8106f30 0x7fe6c8199980 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:39:26.610 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:26.058+0000 7fe6ced60700 1 -- 172.21.15.22:0/1387499479 >> 172.21.15.22:0/1387499479 conn(0x7fe6c81009c0 msgr2=0x7fe6c8100da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:39:26.610 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:26.058+0000 7fe6ced60700 1 -- 172.21.15.22:0/1387499479 shutdown_connections 2024-03-24T09:39:26.610 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:26.058+0000 7fe6ced60700 1 -- 172.21.15.22:0/1387499479 wait complete. 2024-03-24T09:39:26.610 INFO:teuthology.orchestra.run.smithi022.stdout:mon is available 2024-03-24T09:39:26.610 INFO:teuthology.orchestra.run.smithi022.stdout:Assimilating anything we can from ceph.conf... 2024-03-24T09:39:27.595 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout 2024-03-24T09:39:27.595 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout [global] 2024-03-24T09:39:27.595 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout fsid = 45356448-e9c2-11ee-95cd-87774f69a715 2024-03-24T09:39:27.595 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.22:3300,v1:172.21.15.22:6789] 2024-03-24T09:39:27.595 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-24T09:39:27.595 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-24T09:39:27.595 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-24T09:39:27.595 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-24T09:39:27.595 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout 2024-03-24T09:39:27.595 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-24T09:39:27.595 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-24T09:39:27.595 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout 2024-03-24T09:39:27.596 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout [osd] 2024-03-24T09:39:27.596 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-24T09:39:27.596 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-24T09:39:27.596 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-24T09:39:27.596 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-24T09:39:27.596 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-24T09:39:27.596 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.023+0000 7fb3ab792700 1 Processor -- start 2024-03-24T09:39:27.596 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.023+0000 7fb3ab792700 1 -- start start 2024-03-24T09:39:27.596 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.024+0000 7fb3ab792700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb3a4104d90 0x7fb3a41051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:39:27.596 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.024+0000 7fb3ab792700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a4079e30 con 0x7fb3a4104d90 2024-03-24T09:39:27.596 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.024+0000 7fb3a952e700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb3a4104d90 0x7fb3a41051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:39:27.596 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.024+0000 7fb3a952e700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb3a4104d90 0x7fb3a41051b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:53376/0 (socket says 172.21.15.22:53376) 2024-03-24T09:39:27.596 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.024+0000 7fb3a952e700 1 -- 172.21.15.22:0/3896546113 learned_addr learned my addr 172.21.15.22:0/3896546113 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:39:27.596 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.024+0000 7fb3a952e700 1 -- 172.21.15.22:0/3896546113 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a41056f0 con 0x7fb3a4104d90 2024-03-24T09:39:27.596 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.024+0000 7fb3a952e700 1 --2- 172.21.15.22:0/3896546113 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb3a4104d90 0x7fb3a41051b0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fb3a00098a0 tx=0x7fb3a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7bb5e0f9d4e7c58 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:39:27.597 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.024+0000 7fb39bfff700 1 -- 172.21.15.22:0/3896546113 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3a0004180 con 0x7fb3a4104d90 2024-03-24T09:39:27.597 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.024+0000 7fb39bfff700 1 -- 172.21.15.22:0/3896546113 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3a00042e0 con 0x7fb3a4104d90 2024-03-24T09:39:27.597 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.024+0000 7fb39bfff700 1 -- 172.21.15.22:0/3896546113 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3a00036a0 con 0x7fb3a4104d90 2024-03-24T09:39:27.597 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.025+0000 7fb3ab792700 1 -- 172.21.15.22:0/3896546113 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb3a4104d90 msgr2=0x7fb3a41051b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:39:27.597 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.025+0000 7fb3ab792700 1 --2- 172.21.15.22:0/3896546113 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb3a4104d90 0x7fb3a41051b0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fb3a00098a0 tx=0x7fb3a0009bb0 comp rx=0 tx=0).stop 2024-03-24T09:39:27.597 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.025+0000 7fb3ab792700 1 -- 172.21.15.22:0/3896546113 shutdown_connections 2024-03-24T09:39:27.597 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.025+0000 7fb3ab792700 1 --2- 172.21.15.22:0/3896546113 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb3a4104d90 0x7fb3a41051b0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:39:27.597 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.025+0000 7fb3ab792700 1 -- 172.21.15.22:0/3896546113 >> 172.21.15.22:0/3896546113 conn(0x7fb3a4100950 msgr2=0x7fb3a4102db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:39:27.597 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.025+0000 7fb3ab792700 1 -- 172.21.15.22:0/3896546113 shutdown_connections 2024-03-24T09:39:27.597 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.025+0000 7fb3ab792700 1 -- 172.21.15.22:0/3896546113 wait complete. 2024-03-24T09:39:27.597 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.026+0000 7fb3ab792700 1 Processor -- start 2024-03-24T09:39:27.597 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.026+0000 7fb3ab792700 1 -- start start 2024-03-24T09:39:27.597 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.026+0000 7fb3ab792700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb3a4104d90 0x7fb3a419bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:39:27.597 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.026+0000 7fb3ab792700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a419c070 con 0x7fb3a4104d90 2024-03-24T09:39:27.598 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.026+0000 7fb3a952e700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb3a4104d90 0x7fb3a419bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:39:27.598 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.026+0000 7fb3a952e700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb3a4104d90 0x7fb3a419bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:53392/0 (socket says 172.21.15.22:53392) 2024-03-24T09:39:27.598 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.026+0000 7fb3a952e700 1 -- 172.21.15.22:0/4068092915 learned_addr learned my addr 172.21.15.22:0/4068092915 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:39:27.598 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.026+0000 7fb3a952e700 1 -- 172.21.15.22:0/4068092915 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a0009550 con 0x7fb3a4104d90 2024-03-24T09:39:27.598 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.026+0000 7fb3a952e700 1 --2- 172.21.15.22:0/4068092915 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb3a4104d90 0x7fb3a419bb30 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4079270 tx=0x7fb3a0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:39:27.598 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.026+0000 7fb39a7fc700 1 -- 172.21.15.22:0/4068092915 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3a0003ce0 con 0x7fb3a4104d90 2024-03-24T09:39:27.598 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.026+0000 7fb39a7fc700 1 -- 172.21.15.22:0/4068092915 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3a0020070 con 0x7fb3a4104d90 2024-03-24T09:39:27.598 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.027+0000 7fb3ab792700 1 -- 172.21.15.22:0/4068092915 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3a419c270 con 0x7fb3a4104d90 2024-03-24T09:39:27.598 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.027+0000 7fb39a7fc700 1 -- 172.21.15.22:0/4068092915 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3a0011630 con 0x7fb3a4104d90 2024-03-24T09:39:27.598 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.027+0000 7fb3ab792700 1 -- 172.21.15.22:0/4068092915 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3a419c710 con 0x7fb3a4104d90 2024-03-24T09:39:27.598 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.027+0000 7fb39a7fc700 1 -- 172.21.15.22:0/4068092915 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb3a0022030 con 0x7fb3a4104d90 2024-03-24T09:39:27.598 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.027+0000 7fb39a7fc700 1 -- 172.21.15.22:0/4068092915 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb3a0025d00 con 0x7fb3a4104d90 2024-03-24T09:39:27.598 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.028+0000 7fb3ab792700 1 -- 172.21.15.22:0/4068092915 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3a4105a60 con 0x7fb3a4104d90 2024-03-24T09:39:27.599 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.030+0000 7fb39a7fc700 1 -- 172.21.15.22:0/4068092915 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fb3a0025480 con 0x7fb3a4104d90 2024-03-24T09:39:27.599 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.100+0000 7fb3ab792700 1 -- 172.21.15.22:0/4068092915 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fb3a4105bf0 con 0x7fb3a4104d90 2024-03-24T09:39:27.599 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.103+0000 7fb39a7fc700 1 -- 172.21.15.22:0/4068092915 <== mon.0 v2:172.21.15.22:3300/0 7 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fb3a0011c20 con 0x7fb3a4104d90 2024-03-24T09:39:27.599 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.103+0000 7fb39a7fc700 1 -- 172.21.15.22:0/4068092915 <== mon.0 v2:172.21.15.22:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7fb3a00385b0 con 0x7fb3a4104d90 2024-03-24T09:39:27.599 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.104+0000 7fb3ab792700 1 -- 172.21.15.22:0/4068092915 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb3a4104d90 msgr2=0x7fb3a419bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:39:27.599 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.104+0000 7fb3ab792700 1 --2- 172.21.15.22:0/4068092915 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb3a4104d90 0x7fb3a419bb30 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4079270 tx=0x7fb3a0004750 comp rx=0 tx=0).stop 2024-03-24T09:39:27.599 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.104+0000 7fb3ab792700 1 -- 172.21.15.22:0/4068092915 shutdown_connections 2024-03-24T09:39:27.599 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.104+0000 7fb3ab792700 1 --2- 172.21.15.22:0/4068092915 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb3a4104d90 0x7fb3a419bb30 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:39:27.599 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.104+0000 7fb3ab792700 1 -- 172.21.15.22:0/4068092915 >> 172.21.15.22:0/4068092915 conn(0x7fb3a4100950 msgr2=0x7fb3a4192350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:39:27.599 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.104+0000 7fb3ab792700 1 -- 172.21.15.22:0/4068092915 shutdown_connections 2024-03-24T09:39:27.599 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.104+0000 7fb3ab792700 1 -- 172.21.15.22:0/4068092915 wait complete. 2024-03-24T09:39:27.599 INFO:teuthology.orchestra.run.smithi022.stdout:Generating new minimal ceph.conf... 2024-03-24T09:39:28.584 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.961+0000 7fa587024700 1 Processor -- start 2024-03-24T09:39:28.584 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.961+0000 7fa587024700 1 -- start start 2024-03-24T09:39:28.584 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.961+0000 7fa587024700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa580106f70 0x7fa580109360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:39:28.584 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.961+0000 7fa587024700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5800745b0 con 0x7fa580106f70 2024-03-24T09:39:28.584 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.962+0000 7fa584dc0700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa580106f70 0x7fa580109360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:39:28.584 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.962+0000 7fa584dc0700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa580106f70 0x7fa580109360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:53396/0 (socket says 172.21.15.22:53396) 2024-03-24T09:39:28.584 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.962+0000 7fa584dc0700 1 -- 172.21.15.22:0/2425449071 learned_addr learned my addr 172.21.15.22:0/2425449071 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:39:28.585 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.962+0000 7fa584dc0700 1 -- 172.21.15.22:0/2425449071 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5801098a0 con 0x7fa580106f70 2024-03-24T09:39:28.585 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.962+0000 7fa584dc0700 1 --2- 172.21.15.22:0/2425449071 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa580106f70 0x7fa580109360 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa5740098a0 tx=0x7fa574009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc8656b1dd3587bc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:39:28.585 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.962+0000 7fa57f7fe700 1 -- 172.21.15.22:0/2425449071 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa574004180 con 0x7fa580106f70 2024-03-24T09:39:28.585 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.962+0000 7fa57f7fe700 1 -- 172.21.15.22:0/2425449071 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fa5740042e0 con 0x7fa580106f70 2024-03-24T09:39:28.585 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.962+0000 7fa57f7fe700 1 -- 172.21.15.22:0/2425449071 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa574011a70 con 0x7fa580106f70 2024-03-24T09:39:28.585 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.963+0000 7fa587024700 1 -- 172.21.15.22:0/2425449071 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa580106f70 msgr2=0x7fa580109360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:39:28.585 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.963+0000 7fa587024700 1 --2- 172.21.15.22:0/2425449071 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa580106f70 0x7fa580109360 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa5740098a0 tx=0x7fa574009bb0 comp rx=0 tx=0).stop 2024-03-24T09:39:28.585 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.963+0000 7fa587024700 1 -- 172.21.15.22:0/2425449071 shutdown_connections 2024-03-24T09:39:28.585 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.963+0000 7fa587024700 1 --2- 172.21.15.22:0/2425449071 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa580106f70 0x7fa580109360 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:39:28.585 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.963+0000 7fa587024700 1 -- 172.21.15.22:0/2425449071 >> 172.21.15.22:0/2425449071 conn(0x7fa5801009c0 msgr2=0x7fa580102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:39:28.585 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.963+0000 7fa587024700 1 -- 172.21.15.22:0/2425449071 shutdown_connections 2024-03-24T09:39:28.585 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.963+0000 7fa587024700 1 -- 172.21.15.22:0/2425449071 wait complete. 2024-03-24T09:39:28.585 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.963+0000 7fa587024700 1 Processor -- start 2024-03-24T09:39:28.585 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.964+0000 7fa587024700 1 -- start start 2024-03-24T09:39:28.585 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.964+0000 7fa587024700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa580106f70 0x7fa58019a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:39:28.585 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.964+0000 7fa587024700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa58019a720 con 0x7fa580106f70 2024-03-24T09:39:28.586 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.964+0000 7fa584dc0700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa580106f70 0x7fa58019a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:39:28.586 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.964+0000 7fa584dc0700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa580106f70 0x7fa58019a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:53402/0 (socket says 172.21.15.22:53402) 2024-03-24T09:39:28.586 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.964+0000 7fa584dc0700 1 -- 172.21.15.22:0/2634891766 learned_addr learned my addr 172.21.15.22:0/2634891766 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:39:28.586 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.964+0000 7fa584dc0700 1 -- 172.21.15.22:0/2634891766 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa574009550 con 0x7fa580106f70 2024-03-24T09:39:28.586 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.964+0000 7fa584dc0700 1 --2- 172.21.15.22:0/2634891766 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa580106f70 0x7fa58019a1e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa574011a20 tx=0x7fa5740047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:39:28.586 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.964+0000 7fa57dffb700 1 -- 172.21.15.22:0/2634891766 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa574003b10 con 0x7fa580106f70 2024-03-24T09:39:28.586 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.964+0000 7fa57dffb700 1 -- 172.21.15.22:0/2634891766 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fa57401b8c0 con 0x7fa580106f70 2024-03-24T09:39:28.586 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.964+0000 7fa587024700 1 -- 172.21.15.22:0/2634891766 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa58019a920 con 0x7fa580106f70 2024-03-24T09:39:28.586 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.964+0000 7fa57dffb700 1 -- 172.21.15.22:0/2634891766 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5740237b0 con 0x7fa580106f70 2024-03-24T09:39:28.586 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.965+0000 7fa587024700 1 -- 172.21.15.22:0/2634891766 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa58019adc0 con 0x7fa580106f70 2024-03-24T09:39:28.586 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.965+0000 7fa57dffb700 1 -- 172.21.15.22:0/2634891766 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa574023ce0 con 0x7fa580106f70 2024-03-24T09:39:28.586 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.965+0000 7fa57dffb700 1 -- 172.21.15.22:0/2634891766 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa574019710 con 0x7fa580106f70 2024-03-24T09:39:28.586 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.965+0000 7fa587024700 1 -- 172.21.15.22:0/2634891766 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa58010a5f0 con 0x7fa580106f70 2024-03-24T09:39:28.586 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:27.968+0000 7fa57dffb700 1 -- 172.21.15.22:0/2634891766 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fa57401ba30 con 0x7fa580106f70 2024-03-24T09:39:28.587 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:28.038+0000 7fa587024700 1 -- 172.21.15.22:0/2634891766 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fa58019dd40 con 0x7fa580106f70 2024-03-24T09:39:28.587 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:28.039+0000 7fa57dffb700 1 -- 172.21.15.22:0/2634891766 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7fa57401bbd0 con 0x7fa580106f70 2024-03-24T09:39:28.587 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:28.040+0000 7fa587024700 1 -- 172.21.15.22:0/2634891766 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa580106f70 msgr2=0x7fa58019a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:39:28.587 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:28.040+0000 7fa587024700 1 --2- 172.21.15.22:0/2634891766 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa580106f70 0x7fa58019a1e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa574011a20 tx=0x7fa5740047c0 comp rx=0 tx=0).stop 2024-03-24T09:39:28.587 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:28.040+0000 7fa587024700 1 -- 172.21.15.22:0/2634891766 shutdown_connections 2024-03-24T09:39:28.587 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:28.040+0000 7fa587024700 1 --2- 172.21.15.22:0/2634891766 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa580106f70 0x7fa58019a1e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:39:28.587 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:28.040+0000 7fa587024700 1 -- 172.21.15.22:0/2634891766 >> 172.21.15.22:0/2634891766 conn(0x7fa5801009c0 msgr2=0x7fa5801014d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:39:28.587 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:28.040+0000 7fa587024700 1 -- 172.21.15.22:0/2634891766 shutdown_connections 2024-03-24T09:39:28.587 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:28.040+0000 7fa587024700 1 -- 172.21.15.22:0/2634891766 wait complete. 2024-03-24T09:39:28.587 INFO:teuthology.orchestra.run.smithi022.stdout:Restarting the monitor... 2024-03-24T09:39:28.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:28 smithi022 systemd[1]: Stopping Ceph mon.smithi022 for 45356448-e9c2-11ee-95cd-87774f69a715... 2024-03-24T09:39:29.206 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29131]: 2024-03-24T09:39:28.821+0000 7f8be2393700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi022 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T09:39:29.206 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29131]: 2024-03-24T09:39:28.821+0000 7f8be2393700 -1 mon.smithi022@0(leader) e1 *** Got Signal Terminated *** 2024-03-24T09:39:29.206 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:28 smithi022 podman[29390]: 2024-03-24 09:39:28.996345289 +0000 UTC m=+0.256510759 container died aae570ad23da17fce7c5772390c8e2dbd508cb933d2eb5cf974ddefb1a65f6a2 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=reef-738c2fd, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5) 2024-03-24T09:39:29.489 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:29 smithi022 podman[29390]: 2024-03-24 09:39:29.305826183 +0000 UTC m=+0.565991662 container cleanup aae570ad23da17fce7c5772390c8e2dbd508cb933d2eb5cf974ddefb1a65f6a2 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=reef-738c2fd, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-24T09:39:29.489 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:29 smithi022 bash[29390]: ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022 2024-03-24T09:39:29.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:29 smithi022 podman[29404]: 2024-03-24 09:39:29.489692751 +0000 UTC m=+0.493819094 container remove aae570ad23da17fce7c5772390c8e2dbd508cb933d2eb5cf974ddefb1a65f6a2 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=reef-738c2fd, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-24T09:39:29.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:29 smithi022 systemd[1]: ceph-45356448-e9c2-11ee-95cd-87774f69a715@mon.smithi022.service: Deactivated successfully. 2024-03-24T09:39:29.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:29 smithi022 systemd[1]: Stopped Ceph mon.smithi022 for 45356448-e9c2-11ee-95cd-87774f69a715. 2024-03-24T09:39:29.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:29 smithi022 systemd[1]: Starting Ceph mon.smithi022 for 45356448-e9c2-11ee-95cd-87774f69a715... 2024-03-24T09:39:30.084 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:29 smithi022 podman[29494]: 2024-03-24 09:39:29.82405566 +0000 UTC m=+0.112685767 container create 59736938121b1d44b215afbc87b4b244b17d77969e70b1ae08d6bab98c5642fd (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=reef-738c2fd, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T09:39:30.084 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:29 smithi022 podman[29494]: 2024-03-24 09:39:29.729069574 +0000 UTC m=+0.017699635 image pull 68aff9aff3b2fe3c16548acc72564779ed9f377b279f3076ce38885b01df0d46 quay.ceph.io/ceph-ci/ceph:reef 2024-03-24T09:39:30.084 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 podman[29494]: 2024-03-24 09:39:30.02636997 +0000 UTC m=+0.315000024 container init 59736938121b1d44b215afbc87b4b244b17d77969e70b1ae08d6bab98c5642fd (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, RELEASE=reef-738c2fd) 2024-03-24T09:39:30.084 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 podman[29494]: 2024-03-24 09:39:30.029180251 +0000 UTC m=+0.317810303 container start 59736938121b1d44b215afbc87b4b244b17d77969e70b1ae08d6bab98c5642fd (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=reef-738c2fd) 2024-03-24T09:39:30.084 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T09:39:30.084 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: ceph version 18.2.2-952-g738c2fd6 (738c2fd67dbc0b17e2e99568a03974712d998229) reef (stable), process ceph-mon, pid 2 2024-03-24T09:39:30.084 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: pidfile_write: ignore empty --pid-file 2024-03-24T09:39:30.085 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: load: jerasure load: lrc 2024-03-24T09:39:30.085 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: RocksDB version: 7.9.2 2024-03-24T09:39:30.085 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Git sha 0 2024-03-24T09:39:30.085 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Compile date 2024-03-23 03:05:26 2024-03-24T09:39:30.085 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: DB SUMMARY 2024-03-24T09:39:30.085 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: DB Session ID: UFKNYZTCL63TKJM171YY 2024-03-24T09:39:30.085 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: CURRENT file: CURRENT 2024-03-24T09:39:30.085 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T09:39:30.085 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-24T09:39:30.085 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi022/store.db dir, Total Num: 1, files: 000008.sst 2024-03-24T09:39:30.085 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi022/store.db: 000009.log size: 98469 ; 2024-03-24T09:39:30.085 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.error_if_exists: 0 2024-03-24T09:39:30.085 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.create_if_missing: 0 2024-03-24T09:39:30.086 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.paranoid_checks: 1 2024-03-24T09:39:30.087 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T09:39:30.087 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T09:39:30.087 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T09:39:30.087 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.env: 0x559355d51720 2024-03-24T09:39:30.087 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T09:39:30.087 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.info_log: 0x5593590a5380 2024-03-24T09:39:30.087 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T09:39:30.088 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.statistics: (nil) 2024-03-24T09:39:30.088 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.use_fsync: 0 2024-03-24T09:39:30.088 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.max_log_file_size: 0 2024-03-24T09:39:30.088 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T09:39:30.088 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T09:39:30.088 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T09:39:30.088 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T09:39:30.088 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.allow_fallocate: 1 2024-03-24T09:39:30.088 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T09:39:30.088 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T09:39:30.088 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.use_direct_reads: 0 2024-03-24T09:39:30.088 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T09:39:30.088 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T09:39:30.088 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.db_log_dir: 2024-03-24T09:39:30.089 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.wal_dir: 2024-03-24T09:39:30.089 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T09:39:30.089 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T09:39:30.089 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T09:39:30.089 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T09:39:30.089 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T09:39:30.089 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T09:39:30.089 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T09:39:30.089 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T09:39:30.089 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.write_buffer_manager: 0x55935831e140 2024-03-24T09:39:30.089 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T09:39:30.089 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T09:39:30.089 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T09:39:30.089 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T09:39:30.090 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T09:39:30.090 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T09:39:30.090 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T09:39:30.090 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T09:39:30.090 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.unordered_write: 0 2024-03-24T09:39:30.090 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T09:39:30.090 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T09:39:30.090 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T09:39:30.090 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T09:39:30.090 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.row_cache: None 2024-03-24T09:39:30.090 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.wal_filter: None 2024-03-24T09:39:30.090 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T09:39:30.090 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T09:39:30.091 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.two_write_queues: 0 2024-03-24T09:39:30.091 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T09:39:30.091 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.wal_compression: 0 2024-03-24T09:39:30.091 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.atomic_flush: 0 2024-03-24T09:39:30.091 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T09:39:30.091 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T09:39:30.091 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T09:39:30.091 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.log_readahead_size: 0 2024-03-24T09:39:30.091 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T09:39:30.091 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T09:39:30.091 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T09:39:30.091 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T09:39:30.091 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T09:39:30.092 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T09:39:30.092 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T09:39:30.092 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.max_background_jobs: 2 2024-03-24T09:39:30.092 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.max_background_compactions: -1 2024-03-24T09:39:30.092 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.max_subcompactions: 1 2024-03-24T09:39:30.092 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T09:39:30.092 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T09:39:30.092 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T09:39:30.092 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T09:39:30.092 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T09:39:30.092 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T09:39:30.092 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T09:39:30.092 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T09:39:30.092 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.max_open_files: -1 2024-03-24T09:39:30.093 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T09:39:30.093 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T09:39:30.093 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T09:39:30.093 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T09:39:30.093 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.max_background_flushes: -1 2024-03-24T09:39:30.093 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Compression algorithms supported: 2024-03-24T09:39:30.093 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T09:39:30.093 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: kZSTD supported: 0 2024-03-24T09:39:30.093 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: kXpressCompression supported: 0 2024-03-24T09:39:30.093 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T09:39:30.093 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: kZlibCompression supported: 1 2024-03-24T09:39:30.093 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: kSnappyCompression supported: 1 2024-03-24T09:39:30.093 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: kLZ4Compression supported: 1 2024-03-24T09:39:30.093 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: kBZip2Compression supported: 0 2024-03-24T09:39:30.094 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T09:39:30.094 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T09:39:30.094 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi022/store.db/MANIFEST-000010 2024-03-24T09:39:30.094 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T09:39:30.094 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T09:39:30.094 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.merge_operator: 2024-03-24T09:39:30.094 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.compaction_filter: None 2024-03-24T09:39:30.094 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T09:39:30.094 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T09:39:30.094 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T09:39:30.094 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T09:39:30.094 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5593590a54a0) 2024-03-24T09:39:30.094 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: cache_index_and_filter_blocks: 1 2024-03-24T09:39:30.094 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T09:39:30.095 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T09:39:30.095 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: pin_top_level_index_and_filter: 1 2024-03-24T09:39:30.095 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: index_type: 0 2024-03-24T09:39:30.095 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: data_block_index_type: 0 2024-03-24T09:39:30.095 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: index_shortening: 1 2024-03-24T09:39:30.095 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T09:39:30.095 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: checksum: 4 2024-03-24T09:39:30.095 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: no_block_cache: 0 2024-03-24T09:39:30.095 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: block_cache: 0x5593583bf1f0 2024-03-24T09:39:30.095 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: block_cache_name: BinnedLRUCache 2024-03-24T09:39:30.095 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: block_cache_options: 2024-03-24T09:39:30.095 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: capacity : 536870912 2024-03-24T09:39:30.095 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: num_shard_bits : 4 2024-03-24T09:39:30.095 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: strict_capacity_limit : 0 2024-03-24T09:39:30.095 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: high_pri_pool_ratio: 0.000 2024-03-24T09:39:30.095 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: block_cache_compressed: (nil) 2024-03-24T09:39:30.096 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: persistent_cache: (nil) 2024-03-24T09:39:30.096 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: block_size: 4096 2024-03-24T09:39:30.096 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: block_size_deviation: 10 2024-03-24T09:39:30.096 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: block_restart_interval: 16 2024-03-24T09:39:30.096 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: index_block_restart_interval: 1 2024-03-24T09:39:30.096 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: metadata_block_size: 4096 2024-03-24T09:39:30.096 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: partition_filters: 0 2024-03-24T09:39:30.096 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: use_delta_encoding: 1 2024-03-24T09:39:30.096 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: filter_policy: bloomfilter 2024-03-24T09:39:30.096 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: whole_key_filtering: 1 2024-03-24T09:39:30.096 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: verify_compression: 0 2024-03-24T09:39:30.096 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: read_amp_bytes_per_bit: 0 2024-03-24T09:39:30.096 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: format_version: 5 2024-03-24T09:39:30.096 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: enable_index_compression: 1 2024-03-24T09:39:30.097 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: block_align: 0 2024-03-24T09:39:30.097 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: max_auto_readahead_size: 262144 2024-03-24T09:39:30.097 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: prepopulate_block_cache: 0 2024-03-24T09:39:30.097 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: initial_auto_readahead_size: 8192 2024-03-24T09:39:30.097 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T09:39:30.097 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T09:39:30.097 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T09:39:30.097 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.compression: NoCompression 2024-03-24T09:39:30.097 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T09:39:30.097 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T09:39:30.097 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T09:39:30.097 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.num_levels: 7 2024-03-24T09:39:30.097 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T09:39:30.097 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T09:39:30.097 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T09:39:30.097 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T09:39:30.098 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T09:39:30.098 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T09:39:30.098 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T09:39:30.098 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T09:39:30.098 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T09:39:30.098 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T09:39:30.098 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T09:39:30.098 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T09:39:30.098 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T09:39:30.098 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T09:39:30.098 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T09:39:30.098 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T09:39:30.098 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T09:39:30.098 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T09:39:30.099 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T09:39:30.099 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T09:39:30.099 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T09:39:30.099 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T09:39:30.099 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T09:39:30.099 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T09:39:30.099 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T09:39:30.099 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T09:39:30.099 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T09:39:30.099 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T09:39:30.099 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T09:39:30.099 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T09:39:30.099 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T09:39:30.100 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T09:39:30.100 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T09:39:30.100 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T09:39:30.100 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T09:39:30.100 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T09:39:30.100 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T09:39:30.100 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T09:39:30.100 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T09:39:30.100 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T09:39:30.100 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T09:39:30.100 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T09:39:30.100 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T09:39:30.100 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T09:39:30.100 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T09:39:30.100 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T09:39:30.101 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T09:39:30.101 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T09:39:30.101 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T09:39:30.101 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T09:39:30.101 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T09:39:30.101 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T09:39:30.101 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T09:39:30.101 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.table_properties_collectors: 2024-03-24T09:39:30.101 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.inplace_update_support: 0 2024-03-24T09:39:30.101 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T09:39:30.101 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T09:39:30.101 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T09:39:30.101 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T09:39:30.101 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.bloom_locality: 0 2024-03-24T09:39:30.102 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.max_successive_merges: 0 2024-03-24T09:39:30.102 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T09:39:30.102 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T09:39:30.102 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T09:39:30.102 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T09:39:30.102 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.ttl: 2592000 2024-03-24T09:39:30.102 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T09:39:30.102 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T09:39:30.102 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T09:39:30.102 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.enable_blob_files: false 2024-03-24T09:39:30.102 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.min_blob_size: 0 2024-03-24T09:39:30.102 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T09:39:30.102 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T09:39:30.102 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T09:39:30.103 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T09:39:30.103 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T09:39:30.103 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T09:39:30.103 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T09:39:30.103 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T09:39:30.103 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi022/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-24T09:39:30.103 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-24T09:39:30.103 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 2d5b912a-e476-4a4b-9ae6-b823ff51c76a 2024-03-24T09:39:30.103 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711273170056934, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-24T09:39:30.103 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-24T09:39:30.103 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711273170057931, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 93746, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 314, "table_properties": {"data_size": 91780, "index_size": 248, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13576, "raw_average_key_size": 49, "raw_value_size": 84344, "raw_average_value_size": 308, "num_data_blocks": 11, "num_entries": 273, "num_filter_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711273170, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "2d5b912a-e476-4a4b-9ae6-b823ff51c76a", "db_session_id": "UFKNYZTCL63TKJM171YY", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-24T09:39:30.103 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711273170058014, "job": 1, "event": "recovery_finished"} 2024-03-24T09:39:30.103 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-24T09:39:30.103 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi022/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T09:39:30.104 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55935845c000 2024-03-24T09:39:30.104 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: DB pointer 0x55935844a000 2024-03-24T09:39:30.104 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T09:39:30.104 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T09:39:30.104 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: ** DB Stats ** 2024-03-24T09:39:30.104 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T09:39:30.104 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T09:39:30.104 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T09:39:30.104 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T09:39:30.104 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T09:39:30.104 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T09:39:30.104 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T09:39:30.104 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: 2024-03-24T09:39:30.104 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: ** Compaction Stats [default] ** 2024-03-24T09:39:30.105 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T09:39:30.105 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T09:39:30.105 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: L0 2/0 93.42 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T09:39:30.105 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Sum 2/0 93.42 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T09:39:30.105 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T09:39:30.105 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: 2024-03-24T09:39:30.105 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: ** Compaction Stats [default] ** 2024-03-24T09:39:30.105 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T09:39:30.105 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T09:39:30.105 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 143.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T09:39:30.105 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: 2024-03-24T09:39:30.105 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T09:39:30.105 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: 2024-03-24T09:39:30.105 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T09:39:30.105 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T09:39:30.106 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T09:39:30.106 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T09:39:30.106 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T09:39:30.106 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T09:39:30.106 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Cumulative compaction: 0.00 GB write, 29.99 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T09:39:30.106 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Interval compaction: 0.00 GB write, 29.99 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T09:39:30.106 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T09:39:30.106 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Block cache BinnedLRUCache@0x5593583bf1f0#2 capacity: 512.00 MB usage: 1.33 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-24T09:39:30.106 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.44 KB,8.34465e-05%) Misc(1,0.00 KB,0%) 2024-03-24T09:39:30.106 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: 2024-03-24T09:39:30.106 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T09:39:30.106 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: starting mon.smithi022 rank 0 at public addrs [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] at bind addrs [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi022 fsid 45356448-e9c2-11ee-95cd-87774f69a715 2024-03-24T09:39:30.107 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: mon.smithi022@-1(???) e1 preinit fsid 45356448-e9c2-11ee-95cd-87774f69a715 2024-03-24T09:39:30.107 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: mon.smithi022@-1(???).mds e1 new map 2024-03-24T09:39:30.107 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: mon.smithi022@-1(???).mds e1 print_map 2024-03-24T09:39:30.107 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: e1 2024-03-24T09:39:30.107 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T09:39:30.107 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-24T09:39:30.107 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: legacy client fscid: -1 2024-03-24T09:39:30.107 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: 2024-03-24T09:39:30.107 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout: No filesystems configured 2024-03-24T09:39:30.107 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: mon.smithi022@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T09:39:30.107 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: mon.smithi022@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T09:39:30.107 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: mon.smithi022@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T09:39:30.107 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: mon.smithi022@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T09:39:30.108 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: mon.smithi022@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-24T09:39:30.108 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 bash[29494]: 59736938121b1d44b215afbc87b4b244b17d77969e70b1ae08d6bab98c5642fd 2024-03-24T09:39:30.108 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 systemd[1]: Started Ceph mon.smithi022 for 45356448-e9c2-11ee-95cd-87774f69a715. 2024-03-24T09:39:30.108 INFO:teuthology.orchestra.run.smithi022.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-24T09:39:30.395 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: mon.smithi022 is new leader, mons smithi022 in quorum (ranks 0) 2024-03-24T09:39:30.395 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: monmap e1: 1 mons at {smithi022=[v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T09:39:30.395 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: fsmap 2024-03-24T09:39:30.395 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: osdmap e1: 0 total, 0 up, 0 in 2024-03-24T09:39:30.395 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:30 smithi022 ceph-mon[29512]: mgrmap e1: no daemons active 2024-03-24T09:39:31.022 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.498+0000 7f8ddfc3e700 1 Processor -- start 2024-03-24T09:39:31.022 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.498+0000 7f8ddfc3e700 1 -- start start 2024-03-24T09:39:31.022 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.498+0000 7f8ddfc3e700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8dd8104db0 0x7f8dd81051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:39:31.022 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.498+0000 7f8ddfc3e700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd8079e30 con 0x7f8dd8104db0 2024-03-24T09:39:31.022 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.498+0000 7f8ddd9da700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8dd8104db0 0x7f8dd81051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:39:31.022 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.499+0000 7f8ddd9da700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8dd8104db0 0x7f8dd81051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:36932/0 (socket says 172.21.15.22:36932) 2024-03-24T09:39:31.022 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.499+0000 7f8ddd9da700 1 -- 172.21.15.22:0/2173949331 learned_addr learned my addr 172.21.15.22:0/2173949331 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:39:31.022 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.499+0000 7f8ddd9da700 1 -- 172.21.15.22:0/2173949331 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd8105710 con 0x7f8dd8104db0 2024-03-24T09:39:31.022 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.499+0000 7f8ddd9da700 1 --2- 172.21.15.22:0/2173949331 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8dd8104db0 0x7f8dd81051d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f8dd40098a0 tx=0x7f8dd4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9055004f37bce36d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:39:31.023 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.499+0000 7f8ddc9d8700 1 -- 172.21.15.22:0/2173949331 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8dd4004180 con 0x7f8dd8104db0 2024-03-24T09:39:31.023 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.499+0000 7f8ddc9d8700 1 -- 172.21.15.22:0/2173949331 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f8dd40042e0 con 0x7f8dd8104db0 2024-03-24T09:39:31.023 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.499+0000 7f8ddc9d8700 1 -- 172.21.15.22:0/2173949331 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8dd4011a70 con 0x7f8dd8104db0 2024-03-24T09:39:31.023 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.500+0000 7f8ddfc3e700 1 -- 172.21.15.22:0/2173949331 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8dd8104db0 msgr2=0x7f8dd81051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:39:31.023 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.500+0000 7f8ddfc3e700 1 --2- 172.21.15.22:0/2173949331 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8dd8104db0 0x7f8dd81051d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f8dd40098a0 tx=0x7f8dd4009bb0 comp rx=0 tx=0).stop 2024-03-24T09:39:31.023 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.500+0000 7f8ddfc3e700 1 -- 172.21.15.22:0/2173949331 shutdown_connections 2024-03-24T09:39:31.023 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.500+0000 7f8ddfc3e700 1 --2- 172.21.15.22:0/2173949331 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8dd8104db0 0x7f8dd81051d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:39:31.023 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.500+0000 7f8ddfc3e700 1 -- 172.21.15.22:0/2173949331 >> 172.21.15.22:0/2173949331 conn(0x7f8dd81009d0 msgr2=0x7f8dd8102df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:39:31.023 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.500+0000 7f8ddfc3e700 1 -- 172.21.15.22:0/2173949331 shutdown_connections 2024-03-24T09:39:31.023 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.500+0000 7f8ddfc3e700 1 -- 172.21.15.22:0/2173949331 wait complete. 2024-03-24T09:39:31.023 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.500+0000 7f8ddfc3e700 1 Processor -- start 2024-03-24T09:39:31.023 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.500+0000 7f8ddfc3e700 1 -- start start 2024-03-24T09:39:31.023 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.501+0000 7f8ddfc3e700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8dd8104db0 0x7f8dd819c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:39:31.023 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.501+0000 7f8ddfc3e700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd819c910 con 0x7f8dd8104db0 2024-03-24T09:39:31.024 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.501+0000 7f8ddd9da700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8dd8104db0 0x7f8dd819c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:39:31.024 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.501+0000 7f8ddd9da700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8dd8104db0 0x7f8dd819c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:36948/0 (socket says 172.21.15.22:36948) 2024-03-24T09:39:31.024 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.501+0000 7f8ddd9da700 1 -- 172.21.15.22:0/2903080292 learned_addr learned my addr 172.21.15.22:0/2903080292 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:39:31.024 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.501+0000 7f8ddd9da700 1 -- 172.21.15.22:0/2903080292 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd4009550 con 0x7f8dd8104db0 2024-03-24T09:39:31.024 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.501+0000 7f8ddd9da700 1 --2- 172.21.15.22:0/2903080292 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8dd8104db0 0x7f8dd819c3d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8dd4011a20 tx=0x7f8dd40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:39:31.024 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.501+0000 7f8dceffd700 1 -- 172.21.15.22:0/2903080292 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8dd4003b10 con 0x7f8dd8104db0 2024-03-24T09:39:31.024 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.501+0000 7f8dceffd700 1 -- 172.21.15.22:0/2903080292 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f8dd401b8c0 con 0x7f8dd8104db0 2024-03-24T09:39:31.024 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.501+0000 7f8ddfc3e700 1 -- 172.21.15.22:0/2903080292 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dd819cb10 con 0x7f8dd8104db0 2024-03-24T09:39:31.024 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.501+0000 7f8ddfc3e700 1 -- 172.21.15.22:0/2903080292 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dd819cfb0 con 0x7f8dd8104db0 2024-03-24T09:39:31.024 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.501+0000 7f8dceffd700 1 -- 172.21.15.22:0/2903080292 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8dd40237b0 con 0x7f8dd8104db0 2024-03-24T09:39:31.024 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.501+0000 7f8dceffd700 1 -- 172.21.15.22:0/2903080292 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f8dd4023d30 con 0x7f8dd8104db0 2024-03-24T09:39:31.024 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.502+0000 7f8dceffd700 1 -- 172.21.15.22:0/2903080292 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f8dd4019710 con 0x7f8dd8104db0 2024-03-24T09:39:31.024 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.502+0000 7f8ddfc3e700 1 -- 172.21.15.22:0/2903080292 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dd81960e0 con 0x7f8dd8104db0 2024-03-24T09:39:31.024 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.504+0000 7f8dceffd700 1 -- 172.21.15.22:0/2903080292 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f8dd401ba30 con 0x7f8dd8104db0 2024-03-24T09:39:31.025 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.574+0000 7f8ddfc3e700 1 -- 172.21.15.22:0/2903080292 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f8dd802ccb0 con 0x7f8dd8104db0 2024-03-24T09:39:31.025 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.576+0000 7f8dceffd700 1 -- 172.21.15.22:0/2903080292 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f8dd401bbd0 con 0x7f8dd8104db0 2024-03-24T09:39:31.025 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.577+0000 7f8ddfc3e700 1 -- 172.21.15.22:0/2903080292 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8dd8104db0 msgr2=0x7f8dd819c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:39:31.025 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.577+0000 7f8ddfc3e700 1 --2- 172.21.15.22:0/2903080292 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8dd8104db0 0x7f8dd819c3d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8dd4011a20 tx=0x7f8dd40047c0 comp rx=0 tx=0).stop 2024-03-24T09:39:31.025 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.577+0000 7f8ddfc3e700 1 -- 172.21.15.22:0/2903080292 shutdown_connections 2024-03-24T09:39:31.025 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.577+0000 7f8ddfc3e700 1 --2- 172.21.15.22:0/2903080292 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8dd8104db0 0x7f8dd819c3d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:39:31.025 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.577+0000 7f8ddfc3e700 1 -- 172.21.15.22:0/2903080292 >> 172.21.15.22:0/2903080292 conn(0x7f8dd81009d0 msgr2=0x7f8dd8193220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:39:31.025 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.577+0000 7f8ddfc3e700 1 -- 172.21.15.22:0/2903080292 shutdown_connections 2024-03-24T09:39:31.025 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:30.577+0000 7f8ddfc3e700 1 -- 172.21.15.22:0/2903080292 wait complete. 2024-03-24T09:39:31.025 INFO:teuthology.orchestra.run.smithi022.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-24T09:39:31.039 INFO:teuthology.orchestra.run.smithi022.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-24T09:39:31.040 INFO:teuthology.orchestra.run.smithi022.stdout:Creating mgr... 2024-03-24T09:39:31.040 INFO:teuthology.orchestra.run.smithi022.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-24T09:39:31.040 INFO:teuthology.orchestra.run.smithi022.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-24T09:39:31.041 INFO:teuthology.orchestra.run.smithi022.stdout:Verifying port 0.0.0.0:8443 ... 2024-03-24T09:39:31.295 INFO:teuthology.orchestra.run.smithi022.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-45356448-e9c2-11ee-95cd-87774f69a715@mgr.smithi022.emgmxz 2024-03-24T09:39:31.295 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stderr Failed to reset failed state of unit ceph-45356448-e9c2-11ee-95cd-87774f69a715@mgr.smithi022.emgmxz.service: Unit ceph-45356448-e9c2-11ee-95cd-87774f69a715@mgr.smithi022.emgmxz.service not loaded. 2024-03-24T09:39:31.439 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-45356448-e9c2-11ee-95cd-87774f69a715.target.wants/ceph-45356448-e9c2-11ee-95cd-87774f69a715@mgr.smithi022.emgmxz.service → /etc/systemd/system/ceph-45356448-e9c2-11ee-95cd-87774f69a715@.service. 2024-03-24T09:39:31.860 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:31 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/2903080292' entity='client.admin' 2024-03-24T09:39:31.948 INFO:teuthology.orchestra.run.smithi022.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T09:39:31.948 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stdout disabled 2024-03-24T09:39:31.955 INFO:teuthology.orchestra.run.smithi022.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T09:39:31.956 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stdout inactive 2024-03-24T09:39:31.956 INFO:teuthology.orchestra.run.smithi022.stdout:firewalld.service is not enabled 2024-03-24T09:39:31.956 INFO:teuthology.orchestra.run.smithi022.stdout:Not possible to enable service . firewalld.service is not available 2024-03-24T09:39:31.964 INFO:teuthology.orchestra.run.smithi022.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T09:39:31.964 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stdout disabled 2024-03-24T09:39:31.971 INFO:teuthology.orchestra.run.smithi022.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T09:39:31.971 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stdout inactive 2024-03-24T09:39:31.971 INFO:teuthology.orchestra.run.smithi022.stdout:firewalld.service is not enabled 2024-03-24T09:39:31.971 INFO:teuthology.orchestra.run.smithi022.stdout:Not possible to open ports <[9283, 8765, 8443]>. firewalld.service is not available 2024-03-24T09:39:31.971 INFO:teuthology.orchestra.run.smithi022.stdout:Waiting for mgr to start... 2024-03-24T09:39:31.971 INFO:teuthology.orchestra.run.smithi022.stdout:Waiting for mgr... 2024-03-24T09:39:32.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:32 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/1401565469' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T09:39:33.069 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout 2024-03-24T09:39:33.072 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout { 2024-03-24T09:39:33.072 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "fsid": "45356448-e9c2-11ee-95cd-87774f69a715", 2024-03-24T09:39:33.072 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T09:39:33.073 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T09:39:33.073 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T09:39:33.073 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T09:39:33.073 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:39:33.073 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T09:39:33.073 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T09:39:33.073 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout 0 2024-03-24T09:39:33.073 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout ], 2024-03-24T09:39:33.074 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T09:39:33.074 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "smithi022" 2024-03-24T09:39:33.074 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout ], 2024-03-24T09:39:33.074 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-24T09:39:33.074 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T09:39:33.074 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T09:39:33.074 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-03-24T09:39:33.074 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T09:39:33.074 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:39:33.074 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T09:39:33.074 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T09:39:33.074 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T09:39:33.074 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T09:39:33.075 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T09:39:33.082 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T09:39:33.082 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T09:39:33.083 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T09:39:33.083 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:39:33.083 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T09:39:33.083 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T09:39:33.083 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T09:39:33.083 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T09:39:33.083 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T09:39:33.083 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T09:39:33.083 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T09:39:33.083 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T09:39:33.083 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T09:39:33.083 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:39:33.083 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T09:39:33.083 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T09:39:33.084 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T09:39:33.084 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T09:39:33.084 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:39:33.084 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T09:39:33.084 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T09:39:33.084 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T09:39:33.084 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T09:39:33.084 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T09:39:33.084 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T09:39:33.084 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T09:39:33.084 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout ], 2024-03-24T09:39:33.084 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T09:39:33.084 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:39:33.084 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T09:39:33.085 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T09:39:33.085 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T09:39:25.502862+0000", 2024-03-24T09:39:33.085 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T09:39:33.085 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:39:33.085 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T09:39:33.085 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout } 2024-03-24T09:39:33.085 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.380+0000 7fa22ef91700 1 Processor -- start 2024-03-24T09:39:33.085 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.380+0000 7fa22ef91700 1 -- start start 2024-03-24T09:39:33.085 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.380+0000 7fa22ef91700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa228106f90 0x7fa228109380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:39:33.085 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.380+0000 7fa22ef91700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa228074590 con 0x7fa228106f90 2024-03-24T09:39:33.085 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.380+0000 7fa22cd2d700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa228106f90 0x7fa228109380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:39:33.085 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.381+0000 7fa22cd2d700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa228106f90 0x7fa228109380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:36970/0 (socket says 172.21.15.22:36970) 2024-03-24T09:39:33.085 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.381+0000 7fa22cd2d700 1 -- 172.21.15.22:0/877833653 learned_addr learned my addr 172.21.15.22:0/877833653 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:39:33.085 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.381+0000 7fa22cd2d700 1 -- 172.21.15.22:0/877833653 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2281098c0 con 0x7fa228106f90 2024-03-24T09:39:33.086 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.381+0000 7fa22cd2d700 1 --2- 172.21.15.22:0/877833653 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa228106f90 0x7fa228109380 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa21c0098a0 tx=0x7fa21c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94bfdec52e77d9b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:39:33.086 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.381+0000 7fa2277fe700 1 -- 172.21.15.22:0/877833653 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa21c004180 con 0x7fa228106f90 2024-03-24T09:39:33.086 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.381+0000 7fa2277fe700 1 -- 172.21.15.22:0/877833653 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fa21c0042e0 con 0x7fa228106f90 2024-03-24T09:39:33.086 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.381+0000 7fa2277fe700 1 -- 172.21.15.22:0/877833653 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa21c011a70 con 0x7fa228106f90 2024-03-24T09:39:33.086 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.382+0000 7fa22ef91700 1 -- 172.21.15.22:0/877833653 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa228106f90 msgr2=0x7fa228109380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:39:33.086 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.382+0000 7fa22ef91700 1 --2- 172.21.15.22:0/877833653 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa228106f90 0x7fa228109380 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa21c0098a0 tx=0x7fa21c009bb0 comp rx=0 tx=0).stop 2024-03-24T09:39:33.086 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.382+0000 7fa22ef91700 1 -- 172.21.15.22:0/877833653 shutdown_connections 2024-03-24T09:39:33.086 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.382+0000 7fa22ef91700 1 --2- 172.21.15.22:0/877833653 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa228106f90 0x7fa228109380 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:39:33.086 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.382+0000 7fa22ef91700 1 -- 172.21.15.22:0/877833653 >> 172.21.15.22:0/877833653 conn(0x7fa2281009c0 msgr2=0x7fa228102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:39:33.086 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.382+0000 7fa22ef91700 1 -- 172.21.15.22:0/877833653 shutdown_connections 2024-03-24T09:39:33.086 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.382+0000 7fa22ef91700 1 -- 172.21.15.22:0/877833653 wait complete. 2024-03-24T09:39:33.086 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.382+0000 7fa22ef91700 1 Processor -- start 2024-03-24T09:39:33.086 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.382+0000 7fa22ef91700 1 -- start start 2024-03-24T09:39:33.087 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.383+0000 7fa22ef91700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa228106f90 0x7fa2281067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:39:33.087 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.383+0000 7fa22ef91700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa228106d00 con 0x7fa228106f90 2024-03-24T09:39:33.087 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.383+0000 7fa22cd2d700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa228106f90 0x7fa2281067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:39:33.087 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.383+0000 7fa22cd2d700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa228106f90 0x7fa2281067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:36976/0 (socket says 172.21.15.22:36976) 2024-03-24T09:39:33.087 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.383+0000 7fa22cd2d700 1 -- 172.21.15.22:0/1401565469 learned_addr learned my addr 172.21.15.22:0/1401565469 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:39:33.087 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.383+0000 7fa22cd2d700 1 -- 172.21.15.22:0/1401565469 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa21c009550 con 0x7fa228106f90 2024-03-24T09:39:33.087 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.383+0000 7fa22cd2d700 1 --2- 172.21.15.22:0/1401565469 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa228106f90 0x7fa2281067c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa21c011710 tx=0x7fa21c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:39:33.087 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.383+0000 7fa225ffb700 1 -- 172.21.15.22:0/1401565469 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa21c003b10 con 0x7fa228106f90 2024-03-24T09:39:33.087 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.383+0000 7fa22ef91700 1 -- 172.21.15.22:0/1401565469 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa228104f40 con 0x7fa228106f90 2024-03-24T09:39:33.087 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.383+0000 7fa225ffb700 1 -- 172.21.15.22:0/1401565469 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fa21c01b8c0 con 0x7fa228106f90 2024-03-24T09:39:33.087 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.383+0000 7fa225ffb700 1 -- 172.21.15.22:0/1401565469 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa21c0197e0 con 0x7fa228106f90 2024-03-24T09:39:33.087 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.384+0000 7fa22ef91700 1 -- 172.21.15.22:0/1401565469 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2281053e0 con 0x7fa228106f90 2024-03-24T09:39:33.088 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.384+0000 7fa225ffb700 1 -- 172.21.15.22:0/1401565469 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa21c0199d0 con 0x7fa228106f90 2024-03-24T09:39:33.088 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.384+0000 7fa225ffb700 1 -- 172.21.15.22:0/1401565469 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa21c01a3d0 con 0x7fa228106f90 2024-03-24T09:39:33.088 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.385+0000 7fa22ef91700 1 -- 172.21.15.22:0/1401565469 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2280623e0 con 0x7fa228106f90 2024-03-24T09:39:33.088 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.387+0000 7fa225ffb700 1 -- 172.21.15.22:0/1401565469 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fa21c024e60 con 0x7fa228106f90 2024-03-24T09:39:33.088 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.460+0000 7fa22ef91700 1 -- 172.21.15.22:0/1401565469 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa228105760 con 0x7fa228106f90 2024-03-24T09:39:33.088 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.461+0000 7fa225ffb700 1 -- 172.21.15.22:0/1401565469 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7fa21c029060 con 0x7fa228106f90 2024-03-24T09:39:33.088 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.462+0000 7fa22ef91700 1 -- 172.21.15.22:0/1401565469 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa228106f90 msgr2=0x7fa2281067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:39:33.088 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.462+0000 7fa22ef91700 1 --2- 172.21.15.22:0/1401565469 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa228106f90 0x7fa2281067c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa21c011710 tx=0x7fa21c0047c0 comp rx=0 tx=0).stop 2024-03-24T09:39:33.088 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.462+0000 7fa22ef91700 1 -- 172.21.15.22:0/1401565469 shutdown_connections 2024-03-24T09:39:33.088 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.462+0000 7fa22ef91700 1 --2- 172.21.15.22:0/1401565469 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa228106f90 0x7fa2281067c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:39:33.088 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.462+0000 7fa22ef91700 1 -- 172.21.15.22:0/1401565469 >> 172.21.15.22:0/1401565469 conn(0x7fa2281009c0 msgr2=0x7fa228101520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:39:33.088 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.462+0000 7fa22ef91700 1 -- 172.21.15.22:0/1401565469 shutdown_connections 2024-03-24T09:39:33.088 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:32.462+0000 7fa22ef91700 1 -- 172.21.15.22:0/1401565469 wait complete. 2024-03-24T09:39:33.089 INFO:teuthology.orchestra.run.smithi022.stdout:mgr not available, waiting (1/15)... 2024-03-24T09:39:36.121 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:36 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/2487576890' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T09:39:36.323 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout 2024-03-24T09:39:36.323 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout { 2024-03-24T09:39:36.324 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "fsid": "45356448-e9c2-11ee-95cd-87774f69a715", 2024-03-24T09:39:36.324 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T09:39:36.324 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T09:39:36.324 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T09:39:36.324 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T09:39:36.324 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:39:36.324 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T09:39:36.324 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T09:39:36.324 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout 0 2024-03-24T09:39:36.324 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout ], 2024-03-24T09:39:36.324 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T09:39:36.324 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "smithi022" 2024-03-24T09:39:36.324 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout ], 2024-03-24T09:39:36.324 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-24T09:39:36.325 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T09:39:36.325 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T09:39:36.325 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-03-24T09:39:36.325 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T09:39:36.325 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:39:36.325 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T09:39:36.325 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T09:39:36.325 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T09:39:36.325 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T09:39:36.325 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T09:39:36.332 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T09:39:36.332 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T09:39:36.332 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T09:39:36.333 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:39:36.333 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T09:39:36.333 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T09:39:36.333 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T09:39:36.333 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T09:39:36.333 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T09:39:36.333 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T09:39:36.333 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T09:39:36.333 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T09:39:36.333 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T09:39:36.333 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:39:36.333 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T09:39:36.333 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T09:39:36.333 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T09:39:36.334 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T09:39:36.334 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:39:36.334 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T09:39:36.334 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T09:39:36.334 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T09:39:36.334 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T09:39:36.334 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T09:39:36.334 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T09:39:36.334 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T09:39:36.334 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout ], 2024-03-24T09:39:36.334 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T09:39:36.334 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:39:36.334 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T09:39:36.334 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T09:39:36.335 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T09:39:25.502862+0000", 2024-03-24T09:39:36.335 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T09:39:36.335 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:39:36.335 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T09:39:36.335 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout } 2024-03-24T09:39:36.335 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.564+0000 7f1e336d8700 1 Processor -- start 2024-03-24T09:39:36.335 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.564+0000 7f1e336d8700 1 -- start start 2024-03-24T09:39:36.335 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.564+0000 7f1e336d8700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1e2c106530 0x7f1e2c106950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:39:36.335 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.564+0000 7f1e336d8700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2c106f20 con 0x7f1e2c106530 2024-03-24T09:39:36.335 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.564+0000 7f1e31474700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1e2c106530 0x7f1e2c106950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:39:36.335 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.564+0000 7f1e31474700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1e2c106530 0x7f1e2c106950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:36990/0 (socket says 172.21.15.22:36990) 2024-03-24T09:39:36.335 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.564+0000 7f1e31474700 1 -- 172.21.15.22:0/1302207071 learned_addr learned my addr 172.21.15.22:0/1302207071 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:39:36.335 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.565+0000 7f1e31474700 1 -- 172.21.15.22:0/1302207071 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e2c107690 con 0x7f1e2c106530 2024-03-24T09:39:36.335 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.565+0000 7f1e31474700 1 --2- 172.21.15.22:0/1302207071 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1e2c106530 0x7f1e2c106950 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1e280098a0 tx=0x7f1e28009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0749b9f2d5b21be server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:39:36.335 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.565+0000 7f1e23fff700 1 -- 172.21.15.22:0/1302207071 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e28004180 con 0x7f1e2c106530 2024-03-24T09:39:36.336 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.565+0000 7f1e23fff700 1 -- 172.21.15.22:0/1302207071 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f1e280042e0 con 0x7f1e2c106530 2024-03-24T09:39:36.336 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.565+0000 7f1e23fff700 1 -- 172.21.15.22:0/1302207071 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e28011a70 con 0x7f1e2c106530 2024-03-24T09:39:36.336 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.565+0000 7f1e336d8700 1 -- 172.21.15.22:0/1302207071 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1e2c106530 msgr2=0x7f1e2c106950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:39:36.336 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.565+0000 7f1e336d8700 1 --2- 172.21.15.22:0/1302207071 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1e2c106530 0x7f1e2c106950 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1e280098a0 tx=0x7f1e28009bb0 comp rx=0 tx=0).stop 2024-03-24T09:39:36.336 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.566+0000 7f1e336d8700 1 -- 172.21.15.22:0/1302207071 shutdown_connections 2024-03-24T09:39:36.336 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.566+0000 7f1e336d8700 1 --2- 172.21.15.22:0/1302207071 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1e2c106530 0x7f1e2c106950 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:39:36.336 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.566+0000 7f1e336d8700 1 -- 172.21.15.22:0/1302207071 >> 172.21.15.22:0/1302207071 conn(0x7f1e2c101b10 msgr2=0x7f1e2c103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:39:36.336 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.566+0000 7f1e336d8700 1 -- 172.21.15.22:0/1302207071 shutdown_connections 2024-03-24T09:39:36.336 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.566+0000 7f1e336d8700 1 -- 172.21.15.22:0/1302207071 wait complete. 2024-03-24T09:39:36.336 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.566+0000 7f1e336d8700 1 Processor -- start 2024-03-24T09:39:36.336 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.566+0000 7f1e336d8700 1 -- start start 2024-03-24T09:39:36.336 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.566+0000 7f1e336d8700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1e2c106530 0x7f1e2c07bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:39:36.336 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.566+0000 7f1e336d8700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2c07bff0 con 0x7f1e2c106530 2024-03-24T09:39:36.336 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.567+0000 7f1e31474700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1e2c106530 0x7f1e2c07bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:39:36.336 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.567+0000 7f1e31474700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1e2c106530 0x7f1e2c07bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:37000/0 (socket says 172.21.15.22:37000) 2024-03-24T09:39:36.337 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.567+0000 7f1e31474700 1 -- 172.21.15.22:0/2487576890 learned_addr learned my addr 172.21.15.22:0/2487576890 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:39:36.337 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.567+0000 7f1e31474700 1 -- 172.21.15.22:0/2487576890 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e28009550 con 0x7f1e2c106530 2024-03-24T09:39:36.337 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.567+0000 7f1e31474700 1 --2- 172.21.15.22:0/2487576890 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1e2c106530 0x7f1e2c07bab0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1e28011710 tx=0x7f1e280047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:39:36.337 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.567+0000 7f1e227fc700 1 -- 172.21.15.22:0/2487576890 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e28003b10 con 0x7f1e2c106530 2024-03-24T09:39:36.337 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.567+0000 7f1e227fc700 1 -- 172.21.15.22:0/2487576890 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f1e2801b8c0 con 0x7f1e2c106530 2024-03-24T09:39:36.337 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.567+0000 7f1e336d8700 1 -- 172.21.15.22:0/2487576890 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e2c07a280 con 0x7f1e2c106530 2024-03-24T09:39:36.337 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.567+0000 7f1e227fc700 1 -- 172.21.15.22:0/2487576890 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e28019770 con 0x7f1e2c106530 2024-03-24T09:39:36.337 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.567+0000 7f1e336d8700 1 -- 172.21.15.22:0/2487576890 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e2c07a720 con 0x7f1e2c106530 2024-03-24T09:39:36.337 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.568+0000 7f1e227fc700 1 -- 172.21.15.22:0/2487576890 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f1e28019d10 con 0x7f1e2c106530 2024-03-24T09:39:36.337 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.568+0000 7f1e227fc700 1 -- 172.21.15.22:0/2487576890 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f1e2801a760 con 0x7f1e2c106530 2024-03-24T09:39:36.337 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.568+0000 7f1e336d8700 1 -- 172.21.15.22:0/2487576890 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e2c0623e0 con 0x7f1e2c106530 2024-03-24T09:39:36.337 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.570+0000 7f1e227fc700 1 -- 172.21.15.22:0/2487576890 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f1e28029060 con 0x7f1e2c106530 2024-03-24T09:39:36.337 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.644+0000 7f1e336d8700 1 -- 172.21.15.22:0/2487576890 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f1e2c10a940 con 0x7f1e2c106530 2024-03-24T09:39:36.337 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.644+0000 7f1e227fc700 1 -- 172.21.15.22:0/2487576890 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f1e2801f070 con 0x7f1e2c106530 2024-03-24T09:39:36.337 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.645+0000 7f1e336d8700 1 -- 172.21.15.22:0/2487576890 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1e2c106530 msgr2=0x7f1e2c07bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:39:36.338 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.645+0000 7f1e336d8700 1 --2- 172.21.15.22:0/2487576890 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1e2c106530 0x7f1e2c07bab0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1e28011710 tx=0x7f1e280047c0 comp rx=0 tx=0).stop 2024-03-24T09:39:36.338 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.645+0000 7f1e336d8700 1 -- 172.21.15.22:0/2487576890 shutdown_connections 2024-03-24T09:39:36.338 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.645+0000 7f1e336d8700 1 --2- 172.21.15.22:0/2487576890 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1e2c106530 0x7f1e2c07bab0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:39:36.338 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.645+0000 7f1e336d8700 1 -- 172.21.15.22:0/2487576890 >> 172.21.15.22:0/2487576890 conn(0x7f1e2c101b10 msgr2=0x7f1e2c1026c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:39:36.338 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.645+0000 7f1e336d8700 1 -- 172.21.15.22:0/2487576890 shutdown_connections 2024-03-24T09:39:36.338 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:35.645+0000 7f1e336d8700 1 -- 172.21.15.22:0/2487576890 wait complete. 2024-03-24T09:39:36.338 INFO:teuthology.orchestra.run.smithi022.stdout:mgr not available, waiting (2/15)... 2024-03-24T09:39:39.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:38 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/3201896331' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T09:39:39.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:38 smithi022 ceph-mon[29512]: Activating manager daemon smithi022.emgmxz 2024-03-24T09:39:39.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:38 smithi022 ceph-mon[29512]: mgrmap e2: smithi022.emgmxz(active, starting, since 0.00390022s) 2024-03-24T09:39:39.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:38 smithi022 ceph-mon[29512]: from='mgr.14100 172.21.15.22:0/1875994099' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T09:39:39.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:38 smithi022 ceph-mon[29512]: from='mgr.14100 172.21.15.22:0/1875994099' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T09:39:39.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:38 smithi022 ceph-mon[29512]: from='mgr.14100 172.21.15.22:0/1875994099' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T09:39:39.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:38 smithi022 ceph-mon[29512]: from='mgr.14100 172.21.15.22:0/1875994099' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mon metadata", "id": "smithi022"}]: dispatch 2024-03-24T09:39:39.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:38 smithi022 ceph-mon[29512]: from='mgr.14100 172.21.15.22:0/1875994099' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mgr metadata", "who": "smithi022.emgmxz", "id": "smithi022.emgmxz"}]: dispatch 2024-03-24T09:39:39.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:38 smithi022 ceph-mon[29512]: Manager daemon smithi022.emgmxz is now available 2024-03-24T09:39:39.395 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout 2024-03-24T09:39:39.395 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout { 2024-03-24T09:39:39.395 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "fsid": "45356448-e9c2-11ee-95cd-87774f69a715", 2024-03-24T09:39:39.396 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T09:39:39.396 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T09:39:39.396 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T09:39:39.396 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T09:39:39.396 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:39:39.396 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T09:39:39.396 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T09:39:39.396 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout 0 2024-03-24T09:39:39.396 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout ], 2024-03-24T09:39:39.396 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T09:39:39.396 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "smithi022" 2024-03-24T09:39:39.396 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout ], 2024-03-24T09:39:39.396 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-24T09:39:39.396 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T09:39:39.396 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T09:39:39.397 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-03-24T09:39:39.397 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T09:39:39.397 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:39:39.397 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T09:39:39.397 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T09:39:39.397 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T09:39:39.397 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T09:39:39.397 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T09:39:39.405 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T09:39:39.405 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T09:39:39.405 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T09:39:39.405 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:39:39.405 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T09:39:39.405 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T09:39:39.405 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T09:39:39.405 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T09:39:39.405 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T09:39:39.405 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T09:39:39.405 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T09:39:39.405 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T09:39:39.406 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T09:39:39.406 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:39:39.406 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T09:39:39.406 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T09:39:39.406 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T09:39:39.406 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T09:39:39.406 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:39:39.406 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T09:39:39.406 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T09:39:39.406 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T09:39:39.406 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T09:39:39.406 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T09:39:39.406 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T09:39:39.407 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T09:39:39.407 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout ], 2024-03-24T09:39:39.407 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T09:39:39.407 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:39:39.407 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T09:39:39.407 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T09:39:39.407 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T09:39:25.502862+0000", 2024-03-24T09:39:39.407 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T09:39:39.407 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:39:39.407 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T09:39:39.407 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout } 2024-03-24T09:39:39.407 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.735+0000 7f9dc37b2700 1 Processor -- start 2024-03-24T09:39:39.407 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.736+0000 7f9dc37b2700 1 -- start start 2024-03-24T09:39:39.407 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.736+0000 7f9dc37b2700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9dbc106fb0 0x7f9dbc1093a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:39:39.407 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.736+0000 7f9dc37b2700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dbc074570 con 0x7f9dbc106fb0 2024-03-24T09:39:39.407 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.736+0000 7f9dc154e700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9dbc106fb0 0x7f9dbc1093a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:39:39.408 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.736+0000 7f9dc154e700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9dbc106fb0 0x7f9dbc1093a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:53726/0 (socket says 172.21.15.22:53726) 2024-03-24T09:39:39.408 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.736+0000 7f9dc154e700 1 -- 172.21.15.22:0/1825405377 learned_addr learned my addr 172.21.15.22:0/1825405377 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:39:39.408 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.736+0000 7f9dc154e700 1 -- 172.21.15.22:0/1825405377 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dbc1098e0 con 0x7f9dbc106fb0 2024-03-24T09:39:39.408 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.737+0000 7f9dc154e700 1 --2- 172.21.15.22:0/1825405377 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9dbc106fb0 0x7f9dbc1093a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9db80098a0 tx=0x7f9db8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52878d7bb896b519 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:39:39.408 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.737+0000 7f9daffff700 1 -- 172.21.15.22:0/1825405377 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9db8004180 con 0x7f9dbc106fb0 2024-03-24T09:39:39.408 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.737+0000 7f9daffff700 1 -- 172.21.15.22:0/1825405377 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f9db80042e0 con 0x7f9dbc106fb0 2024-03-24T09:39:39.408 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.737+0000 7f9daffff700 1 -- 172.21.15.22:0/1825405377 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9db8011a70 con 0x7f9dbc106fb0 2024-03-24T09:39:39.408 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.737+0000 7f9dc37b2700 1 -- 172.21.15.22:0/1825405377 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9dbc106fb0 msgr2=0x7f9dbc1093a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:39:39.408 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.737+0000 7f9dc37b2700 1 --2- 172.21.15.22:0/1825405377 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9dbc106fb0 0x7f9dbc1093a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9db80098a0 tx=0x7f9db8009bb0 comp rx=0 tx=0).stop 2024-03-24T09:39:39.408 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.737+0000 7f9dc37b2700 1 -- 172.21.15.22:0/1825405377 shutdown_connections 2024-03-24T09:39:39.408 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.737+0000 7f9dc37b2700 1 --2- 172.21.15.22:0/1825405377 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9dbc106fb0 0x7f9dbc1093a0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:39:39.408 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.737+0000 7f9dc37b2700 1 -- 172.21.15.22:0/1825405377 >> 172.21.15.22:0/1825405377 conn(0x7f9dbc1009c0 msgr2=0x7f9dbc102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:39:39.408 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.738+0000 7f9dc37b2700 1 -- 172.21.15.22:0/1825405377 shutdown_connections 2024-03-24T09:39:39.408 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.738+0000 7f9dc37b2700 1 -- 172.21.15.22:0/1825405377 wait complete. 2024-03-24T09:39:39.408 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.738+0000 7f9dc37b2700 1 Processor -- start 2024-03-24T09:39:39.409 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.738+0000 7f9dc37b2700 1 -- start start 2024-03-24T09:39:39.409 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.738+0000 7f9dc37b2700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9dbc106fb0 0x7f9dbc19a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:39:39.409 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.738+0000 7f9dc37b2700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dbc19a770 con 0x7f9dbc106fb0 2024-03-24T09:39:39.409 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.738+0000 7f9dc154e700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9dbc106fb0 0x7f9dbc19a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:39:39.409 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.738+0000 7f9dc154e700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9dbc106fb0 0x7f9dbc19a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:53732/0 (socket says 172.21.15.22:53732) 2024-03-24T09:39:39.409 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.738+0000 7f9dc154e700 1 -- 172.21.15.22:0/3201896331 learned_addr learned my addr 172.21.15.22:0/3201896331 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:39:39.409 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.739+0000 7f9dc154e700 1 -- 172.21.15.22:0/3201896331 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db8009550 con 0x7f9dbc106fb0 2024-03-24T09:39:39.409 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.739+0000 7f9dc154e700 1 --2- 172.21.15.22:0/3201896331 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9dbc106fb0 0x7f9dbc19a230 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9db8011a20 tx=0x7f9db80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:39:39.409 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.739+0000 7f9dae7fc700 1 -- 172.21.15.22:0/3201896331 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9db8003b10 con 0x7f9dbc106fb0 2024-03-24T09:39:39.409 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.739+0000 7f9dae7fc700 1 -- 172.21.15.22:0/3201896331 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f9db801b8c0 con 0x7f9dbc106fb0 2024-03-24T09:39:39.409 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.739+0000 7f9dc37b2700 1 -- 172.21.15.22:0/3201896331 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dbc19a970 con 0x7f9dbc106fb0 2024-03-24T09:39:39.409 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.739+0000 7f9dae7fc700 1 -- 172.21.15.22:0/3201896331 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9db80237b0 con 0x7f9dbc106fb0 2024-03-24T09:39:39.409 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.739+0000 7f9dc37b2700 1 -- 172.21.15.22:0/3201896331 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dbc19ae10 con 0x7f9dbc106fb0 2024-03-24T09:39:39.409 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.739+0000 7f9dae7fc700 1 -- 172.21.15.22:0/3201896331 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9db8023ce0 con 0x7f9dbc106fb0 2024-03-24T09:39:39.409 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.739+0000 7f9dae7fc700 1 -- 172.21.15.22:0/3201896331 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9db8019710 con 0x7f9dbc106fb0 2024-03-24T09:39:39.410 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.740+0000 7f9dc37b2700 1 -- 172.21.15.22:0/3201896331 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dbc10a600 con 0x7f9dbc106fb0 2024-03-24T09:39:39.410 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.742+0000 7f9dae7fc700 1 -- 172.21.15.22:0/3201896331 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f9db801ba30 con 0x7f9dbc106fb0 2024-03-24T09:39:39.410 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.815+0000 7f9dc37b2700 1 -- 172.21.15.22:0/3201896331 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f9dbc19ddb0 con 0x7f9dbc106fb0 2024-03-24T09:39:39.410 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.815+0000 7f9dae7fc700 1 -- 172.21.15.22:0/3201896331 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f9db801bbd0 con 0x7f9dbc106fb0 2024-03-24T09:39:39.410 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.816+0000 7f9dc37b2700 1 -- 172.21.15.22:0/3201896331 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9dbc106fb0 msgr2=0x7f9dbc19a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:39:39.410 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.816+0000 7f9dc37b2700 1 --2- 172.21.15.22:0/3201896331 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9dbc106fb0 0x7f9dbc19a230 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9db8011a20 tx=0x7f9db80047c0 comp rx=0 tx=0).stop 2024-03-24T09:39:39.410 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.816+0000 7f9dc37b2700 1 -- 172.21.15.22:0/3201896331 shutdown_connections 2024-03-24T09:39:39.410 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.816+0000 7f9dc37b2700 1 --2- 172.21.15.22:0/3201896331 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9dbc106fb0 0x7f9dbc19a230 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:39:39.410 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.816+0000 7f9dc37b2700 1 -- 172.21.15.22:0/3201896331 >> 172.21.15.22:0/3201896331 conn(0x7f9dbc1009c0 msgr2=0x7f9dbc101550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:39:39.410 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.816+0000 7f9dc37b2700 1 -- 172.21.15.22:0/3201896331 shutdown_connections 2024-03-24T09:39:39.410 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:38.816+0000 7f9dc37b2700 1 -- 172.21.15.22:0/3201896331 wait complete. 2024-03-24T09:39:39.410 INFO:teuthology.orchestra.run.smithi022.stdout:mgr not available, waiting (3/15)... 2024-03-24T09:39:40.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:39 smithi022 ceph-mon[29512]: from='mgr.14100 172.21.15.22:0/1875994099' entity='mgr.smithi022.emgmxz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.emgmxz/mirror_snapshot_schedule"}]: dispatch 2024-03-24T09:39:40.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:39 smithi022 ceph-mon[29512]: from='mgr.14100 172.21.15.22:0/1875994099' entity='mgr.smithi022.emgmxz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.emgmxz/trash_purge_schedule"}]: dispatch 2024-03-24T09:39:40.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:39 smithi022 ceph-mon[29512]: from='mgr.14100 172.21.15.22:0/1875994099' entity='mgr.smithi022.emgmxz' 2024-03-24T09:39:40.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:39 smithi022 ceph-mon[29512]: from='mgr.14100 172.21.15.22:0/1875994099' entity='mgr.smithi022.emgmxz' 2024-03-24T09:39:40.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:39 smithi022 ceph-mon[29512]: from='mgr.14100 172.21.15.22:0/1875994099' entity='mgr.smithi022.emgmxz' 2024-03-24T09:39:40.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:39 smithi022 ceph-mon[29512]: mgrmap e3: smithi022.emgmxz(active, since 1.00725s) 2024-03-24T09:39:42.483 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:42 smithi022 ceph-mon[29512]: mgrmap e4: smithi022.emgmxz(active, since 2s) 2024-03-24T09:39:44.716 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:44 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/664607816' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T09:39:48.747 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout 2024-03-24T09:39:48.747 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout { 2024-03-24T09:39:48.747 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "fsid": "45356448-e9c2-11ee-95cd-87774f69a715", 2024-03-24T09:39:48.747 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T09:39:48.747 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T09:39:48.747 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T09:39:48.747 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T09:39:48.747 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:39:48.747 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T09:39:48.747 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T09:39:48.747 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout 0 2024-03-24T09:39:48.747 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout ], 2024-03-24T09:39:48.748 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T09:39:48.748 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "smithi022" 2024-03-24T09:39:48.748 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout ], 2024-03-24T09:39:48.748 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-03-24T09:39:48.748 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T09:39:48.748 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T09:39:48.748 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-03-24T09:39:48.748 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T09:39:48.748 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:39:48.748 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T09:39:48.748 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T09:39:48.748 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T09:39:48.748 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T09:39:48.748 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T09:39:48.748 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T09:39:48.749 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T09:39:48.749 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T09:39:48.749 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:39:48.749 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T09:39:48.749 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T09:39:48.749 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T09:39:48.749 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T09:39:48.749 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T09:39:48.749 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T09:39:48.749 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T09:39:48.749 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T09:39:48.749 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T09:39:48.749 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:39:48.749 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T09:39:48.750 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T09:39:48.750 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T09:39:48.750 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T09:39:48.750 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:39:48.750 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T09:39:48.750 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T09:39:48.750 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T09:39:48.750 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T09:39:48.750 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T09:39:48.750 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T09:39:48.750 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T09:39:48.750 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout ], 2024-03-24T09:39:48.750 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T09:39:48.750 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:39:48.751 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T09:39:48.751 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T09:39:48.751 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T09:39:25.502862+0000", 2024-03-24T09:39:48.751 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T09:39:48.751 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:39:48.751 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T09:39:48.751 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout } 2024-03-24T09:39:48.751 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.253+0000 7f586d0b3700 1 Processor -- start 2024-03-24T09:39:48.751 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.253+0000 7f586d0b3700 1 -- start start 2024-03-24T09:39:48.751 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.253+0000 7f586d0b3700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5868105940 0x7f5868105d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:39:48.751 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.253+0000 7f586d0b3700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5868106330 con 0x7f5868105940 2024-03-24T09:39:48.751 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.253+0000 7f5866d9d700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5868105940 0x7f5868105d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:39:48.751 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.253+0000 7f5866d9d700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5868105940 0x7f5868105d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:53792/0 (socket says 172.21.15.22:53792) 2024-03-24T09:39:48.751 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.253+0000 7f5866d9d700 1 -- 172.21.15.22:0/352589450 learned_addr learned my addr 172.21.15.22:0/352589450 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:39:48.752 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.253+0000 7f5866d9d700 1 -- 172.21.15.22:0/352589450 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5868106660 con 0x7f5868105940 2024-03-24T09:39:48.752 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.253+0000 7f5866d9d700 1 --2- 172.21.15.22:0/352589450 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5868105940 0x7f5868105d60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f58580098a0 tx=0x7f5858009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdbf40e3377ffc16 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:39:48.752 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.254+0000 7f5865d9b700 1 -- 172.21.15.22:0/352589450 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5858004180 con 0x7f5868105940 2024-03-24T09:39:48.752 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.254+0000 7f5865d9b700 1 -- 172.21.15.22:0/352589450 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f58580042e0 con 0x7f5868105940 2024-03-24T09:39:48.752 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.254+0000 7f5865d9b700 1 -- 172.21.15.22:0/352589450 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5858011a70 con 0x7f5868105940 2024-03-24T09:39:48.752 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.254+0000 7f586d0b3700 1 -- 172.21.15.22:0/352589450 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5868105940 msgr2=0x7f5868105d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:39:48.752 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.254+0000 7f586d0b3700 1 --2- 172.21.15.22:0/352589450 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5868105940 0x7f5868105d60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f58580098a0 tx=0x7f5858009bb0 comp rx=0 tx=0).stop 2024-03-24T09:39:48.752 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.254+0000 7f586d0b3700 1 -- 172.21.15.22:0/352589450 shutdown_connections 2024-03-24T09:39:48.752 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.254+0000 7f586d0b3700 1 --2- 172.21.15.22:0/352589450 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5868105940 0x7f5868105d60 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:39:48.752 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.254+0000 7f586d0b3700 1 -- 172.21.15.22:0/352589450 >> 172.21.15.22:0/352589450 conn(0x7f58681001a0 msgr2=0x7f58681025c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:39:48.752 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.254+0000 7f586d0b3700 1 -- 172.21.15.22:0/352589450 shutdown_connections 2024-03-24T09:39:48.752 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.254+0000 7f586d0b3700 1 -- 172.21.15.22:0/352589450 wait complete. 2024-03-24T09:39:48.752 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.255+0000 7f586d0b3700 1 Processor -- start 2024-03-24T09:39:48.752 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.255+0000 7f586d0b3700 1 -- start start 2024-03-24T09:39:48.752 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.255+0000 7f586d0b3700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5868105940 0x7f5868104ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:39:48.753 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.255+0000 7f586d0b3700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5868105410 con 0x7f5868105940 2024-03-24T09:39:48.753 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.255+0000 7f5866d9d700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5868105940 0x7f5868104ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:39:48.753 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.255+0000 7f5866d9d700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5868105940 0x7f5868104ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:53806/0 (socket says 172.21.15.22:53806) 2024-03-24T09:39:48.753 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.256+0000 7f5866d9d700 1 -- 172.21.15.22:0/664607816 learned_addr learned my addr 172.21.15.22:0/664607816 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:39:48.753 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.256+0000 7f5866d9d700 1 -- 172.21.15.22:0/664607816 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5858009550 con 0x7f5868105940 2024-03-24T09:39:48.753 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.256+0000 7f5866d9d700 1 --2- 172.21.15.22:0/664607816 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5868105940 0x7f5868104ed0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5858011a20 tx=0x7f58580047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:39:48.753 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.256+0000 7f584ffff700 1 -- 172.21.15.22:0/664607816 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5858003b10 con 0x7f5868105940 2024-03-24T09:39:48.753 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.256+0000 7f586d0b3700 1 -- 172.21.15.22:0/664607816 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f586819ad80 con 0x7f5868105940 2024-03-24T09:39:48.753 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.256+0000 7f584ffff700 1 -- 172.21.15.22:0/664607816 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f585801b8c0 con 0x7f5868105940 2024-03-24T09:39:48.753 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.256+0000 7f584ffff700 1 -- 172.21.15.22:0/664607816 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58580237b0 con 0x7f5868105940 2024-03-24T09:39:48.753 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.256+0000 7f586d0b3700 1 -- 172.21.15.22:0/664607816 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f586819f5e0 con 0x7f5868105940 2024-03-24T09:39:48.753 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.257+0000 7f584ffff700 1 -- 172.21.15.22:0/664607816 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 4) v1 ==== 45537+0+0 (secure 0 0 0) 0x7f5858023a30 con 0x7f5868105940 2024-03-24T09:39:48.754 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.257+0000 7f586d0b3700 1 -- 172.21.15.22:0/664607816 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5868109890 con 0x7f5868105940 2024-03-24T09:39:48.754 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.257+0000 7f584ffff700 1 --2- 172.21.15.22:0/664607816 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7f5850038690 0x7f585003ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:39:48.754 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.257+0000 7f584ffff700 1 -- 172.21.15.22:0/664607816 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f585804caa0 con 0x7f5868105940 2024-03-24T09:39:48.754 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.258+0000 7f586659c700 1 --2- 172.21.15.22:0/664607816 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7f5850038690 0x7f585003ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:39:48.754 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.258+0000 7f586659c700 1 --2- 172.21.15.22:0/664607816 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7f5850038690 0x7f585003ab50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f585c006fd0 tx=0x7f585c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:39:48.754 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.263+0000 7f584ffff700 1 -- 172.21.15.22:0/664607816 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5858018760 con 0x7f5868105940 2024-03-24T09:39:48.754 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.552+0000 7f586d0b3700 1 -- 172.21.15.22:0/664607816 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f586819fc40 con 0x7f5868105940 2024-03-24T09:39:48.754 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.552+0000 7f584ffff700 1 -- 172.21.15.22:0/664607816 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f585800b3e0 con 0x7f5868105940 2024-03-24T09:39:48.754 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.554+0000 7f586d0b3700 1 -- 172.21.15.22:0/664607816 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7f5850038690 msgr2=0x7f585003ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:39:48.754 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.554+0000 7f586d0b3700 1 --2- 172.21.15.22:0/664607816 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7f5850038690 0x7f585003ab50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f585c006fd0 tx=0x7f585c006e40 comp rx=0 tx=0).stop 2024-03-24T09:39:48.754 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.554+0000 7f586d0b3700 1 -- 172.21.15.22:0/664607816 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5868105940 msgr2=0x7f5868104ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:39:48.754 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.554+0000 7f586d0b3700 1 --2- 172.21.15.22:0/664607816 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5868105940 0x7f5868104ed0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5858011a20 tx=0x7f58580047c0 comp rx=0 tx=0).stop 2024-03-24T09:39:48.754 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.555+0000 7f586d0b3700 1 -- 172.21.15.22:0/664607816 shutdown_connections 2024-03-24T09:39:48.755 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.555+0000 7f586d0b3700 1 --2- 172.21.15.22:0/664607816 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7f5850038690 0x7f585003ab50 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:39:48.755 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.555+0000 7f586d0b3700 1 --2- 172.21.15.22:0/664607816 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5868105940 0x7f5868104ed0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:39:48.755 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.555+0000 7f586d0b3700 1 -- 172.21.15.22:0/664607816 >> 172.21.15.22:0/664607816 conn(0x7f58681001a0 msgr2=0x7f5868100d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:39:48.755 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.555+0000 7f586d0b3700 1 -- 172.21.15.22:0/664607816 shutdown_connections 2024-03-24T09:39:48.755 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:44.555+0000 7f586d0b3700 1 -- 172.21.15.22:0/664607816 wait complete. 2024-03-24T09:39:48.755 INFO:teuthology.orchestra.run.smithi022.stdout:mgr is available 2024-03-24T09:39:49.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:49 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/3428046505' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-24T09:39:49.882 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout 2024-03-24T09:39:49.882 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout [global] 2024-03-24T09:39:49.882 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout fsid = 45356448-e9c2-11ee-95cd-87774f69a715 2024-03-24T09:39:49.882 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-24T09:39:49.883 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-24T09:39:49.883 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-24T09:39:49.883 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-24T09:39:49.883 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout 2024-03-24T09:39:49.883 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-24T09:39:49.883 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-24T09:39:49.883 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout 2024-03-24T09:39:49.883 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout [osd] 2024-03-24T09:39:49.883 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-24T09:39:49.883 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-24T09:39:49.883 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-24T09:39:49.883 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-24T09:39:49.883 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-24T09:39:49.883 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.185+0000 7f2e9e4e2700 1 Processor -- start 2024-03-24T09:39:49.884 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.185+0000 7f2e9e4e2700 1 -- start start 2024-03-24T09:39:49.884 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.186+0000 7f2e9e4e2700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2e9807be10 0x7f2e9807a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:39:49.884 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.186+0000 7f2e9e4e2700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9807a800 con 0x7f2e9807be10 2024-03-24T09:39:49.884 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.186+0000 7f2e97fff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2e9807be10 0x7f2e9807a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:39:49.884 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.186+0000 7f2e97fff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2e9807be10 0x7f2e9807a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:59626/0 (socket says 172.21.15.22:59626) 2024-03-24T09:39:49.884 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.186+0000 7f2e97fff700 1 -- 172.21.15.22:0/1950886357 learned_addr learned my addr 172.21.15.22:0/1950886357 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:39:49.884 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.186+0000 7f2e97fff700 1 -- 172.21.15.22:0/1950886357 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e9807a940 con 0x7f2e9807be10 2024-03-24T09:39:49.884 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.186+0000 7f2e97fff700 1 --2- 172.21.15.22:0/1950886357 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2e9807be10 0x7f2e9807a2c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2e800098a0 tx=0x7f2e80009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5d823726aa14270 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:39:49.884 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.187+0000 7f2e96ffd700 1 -- 172.21.15.22:0/1950886357 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e80004180 con 0x7f2e9807be10 2024-03-24T09:39:49.884 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.187+0000 7f2e96ffd700 1 -- 172.21.15.22:0/1950886357 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f2e800042e0 con 0x7f2e9807be10 2024-03-24T09:39:49.884 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.187+0000 7f2e96ffd700 1 -- 172.21.15.22:0/1950886357 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e80011a70 con 0x7f2e9807be10 2024-03-24T09:39:49.884 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.187+0000 7f2e9e4e2700 1 -- 172.21.15.22:0/1950886357 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2e9807be10 msgr2=0x7f2e9807a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:39:49.884 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.187+0000 7f2e9e4e2700 1 --2- 172.21.15.22:0/1950886357 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2e9807be10 0x7f2e9807a2c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2e800098a0 tx=0x7f2e80009bb0 comp rx=0 tx=0).stop 2024-03-24T09:39:49.884 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.187+0000 7f2e9e4e2700 1 -- 172.21.15.22:0/1950886357 shutdown_connections 2024-03-24T09:39:49.885 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.187+0000 7f2e9e4e2700 1 --2- 172.21.15.22:0/1950886357 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2e9807be10 0x7f2e9807a2c0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:39:49.885 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.187+0000 7f2e9e4e2700 1 -- 172.21.15.22:0/1950886357 >> 172.21.15.22:0/1950886357 conn(0x7f2e98101b10 msgr2=0x7f2e98103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:39:49.885 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.187+0000 7f2e9e4e2700 1 -- 172.21.15.22:0/1950886357 shutdown_connections 2024-03-24T09:39:49.885 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.187+0000 7f2e9e4e2700 1 -- 172.21.15.22:0/1950886357 wait complete. 2024-03-24T09:39:49.885 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.188+0000 7f2e9e4e2700 1 Processor -- start 2024-03-24T09:39:49.885 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.188+0000 7f2e9e4e2700 1 -- start start 2024-03-24T09:39:49.885 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.188+0000 7f2e9e4e2700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2e9807be10 0x7f2e9819c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:39:49.885 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.188+0000 7f2e9e4e2700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9819c970 con 0x7f2e9807be10 2024-03-24T09:39:49.885 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.188+0000 7f2e97fff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2e9807be10 0x7f2e9819c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:39:49.885 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.188+0000 7f2e97fff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2e9807be10 0x7f2e9819c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:59630/0 (socket says 172.21.15.22:59630) 2024-03-24T09:39:49.885 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.188+0000 7f2e97fff700 1 -- 172.21.15.22:0/3428046505 learned_addr learned my addr 172.21.15.22:0/3428046505 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:39:49.885 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.189+0000 7f2e97fff700 1 -- 172.21.15.22:0/3428046505 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e80009550 con 0x7f2e9807be10 2024-03-24T09:39:49.885 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.189+0000 7f2e97fff700 1 --2- 172.21.15.22:0/3428046505 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2e9807be10 0x7f2e9819c430 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2e80011a20 tx=0x7f2e800047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:39:49.885 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.189+0000 7f2e957fa700 1 -- 172.21.15.22:0/3428046505 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e80003b10 con 0x7f2e9807be10 2024-03-24T09:39:49.885 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.189+0000 7f2e957fa700 1 -- 172.21.15.22:0/3428046505 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f2e8001b8c0 con 0x7f2e9807be10 2024-03-24T09:39:49.886 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.189+0000 7f2e9e4e2700 1 -- 172.21.15.22:0/3428046505 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e9819cb70 con 0x7f2e9807be10 2024-03-24T09:39:49.886 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.189+0000 7f2e957fa700 1 -- 172.21.15.22:0/3428046505 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e800237b0 con 0x7f2e9807be10 2024-03-24T09:39:49.886 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.189+0000 7f2e9e4e2700 1 -- 172.21.15.22:0/3428046505 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e9819d010 con 0x7f2e9807be10 2024-03-24T09:39:49.886 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.190+0000 7f2e957fa700 1 -- 172.21.15.22:0/3428046505 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 4) v1 ==== 45537+0+0 (secure 0 0 0) 0x7f2e800239e0 con 0x7f2e9807be10 2024-03-24T09:39:49.886 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.190+0000 7f2e957fa700 1 --2- 172.21.15.22:0/3428046505 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7f2e840386e0 0x7f2e8403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:39:49.886 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.190+0000 7f2e957fa700 1 -- 172.21.15.22:0/3428046505 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2e8004cbe0 con 0x7f2e9807be10 2024-03-24T09:39:49.886 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.190+0000 7f2e977fe700 1 --2- 172.21.15.22:0/3428046505 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7f2e840386e0 0x7f2e8403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:39:49.886 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.190+0000 7f2e9e4e2700 1 -- 172.21.15.22:0/3428046505 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e980623e0 con 0x7f2e9807be10 2024-03-24T09:39:49.886 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.191+0000 7f2e977fe700 1 --2- 172.21.15.22:0/3428046505 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7f2e840386e0 0x7f2e8403aba0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2e88006fd0 tx=0x7f2e88006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:39:49.886 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.195+0000 7f2e957fa700 1 -- 172.21.15.22:0/3428046505 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2e80023c90 con 0x7f2e9807be10 2024-03-24T09:39:49.886 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.397+0000 7f2e9e4e2700 1 -- 172.21.15.22:0/3428046505 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f2e9819fc70 con 0x7f2e9807be10 2024-03-24T09:39:49.886 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.398+0000 7f2e957fa700 1 -- 172.21.15.22:0/3428046505 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f2e8000b3e0 con 0x7f2e9807be10 2024-03-24T09:39:49.887 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.400+0000 7f2e9e4e2700 1 -- 172.21.15.22:0/3428046505 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7f2e840386e0 msgr2=0x7f2e8403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:39:49.887 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.400+0000 7f2e9e4e2700 1 --2- 172.21.15.22:0/3428046505 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7f2e840386e0 0x7f2e8403aba0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2e88006fd0 tx=0x7f2e88006e40 comp rx=0 tx=0).stop 2024-03-24T09:39:49.887 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.400+0000 7f2e9e4e2700 1 -- 172.21.15.22:0/3428046505 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2e9807be10 msgr2=0x7f2e9819c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:39:49.887 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.400+0000 7f2e9e4e2700 1 --2- 172.21.15.22:0/3428046505 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2e9807be10 0x7f2e9819c430 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2e80011a20 tx=0x7f2e800047c0 comp rx=0 tx=0).stop 2024-03-24T09:39:49.887 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.400+0000 7f2e9e4e2700 1 -- 172.21.15.22:0/3428046505 shutdown_connections 2024-03-24T09:39:49.887 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.400+0000 7f2e9e4e2700 1 --2- 172.21.15.22:0/3428046505 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7f2e840386e0 0x7f2e8403aba0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:39:49.887 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.400+0000 7f2e9e4e2700 1 --2- 172.21.15.22:0/3428046505 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2e9807be10 0x7f2e9819c430 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:39:49.887 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.400+0000 7f2e9e4e2700 1 -- 172.21.15.22:0/3428046505 >> 172.21.15.22:0/3428046505 conn(0x7f2e98101b10 msgr2=0x7f2e981026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:39:49.887 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.400+0000 7f2e9e4e2700 1 -- 172.21.15.22:0/3428046505 shutdown_connections 2024-03-24T09:39:49.887 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:49.400+0000 7f2e9e4e2700 1 -- 172.21.15.22:0/3428046505 wait complete. 2024-03-24T09:39:49.887 INFO:teuthology.orchestra.run.smithi022.stdout:Enabling cephadm module... 2024-03-24T09:39:51.385 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:51 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/1649476045' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-24T09:39:51.676 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.291+0000 7f7a8a1ef700 1 Processor -- start 2024-03-24T09:39:51.677 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.291+0000 7f7a8a1ef700 1 -- start start 2024-03-24T09:39:51.677 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.291+0000 7f7a8a1ef700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7a84106550 0x7f7a84106970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:39:51.677 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.291+0000 7f7a8a1ef700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a84106f40 con 0x7f7a84106550 2024-03-24T09:39:51.677 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.291+0000 7f7a837fe700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7a84106550 0x7f7a84106970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:39:51.677 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.291+0000 7f7a837fe700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7a84106550 0x7f7a84106970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:59646/0 (socket says 172.21.15.22:59646) 2024-03-24T09:39:51.677 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.291+0000 7f7a837fe700 1 -- 172.21.15.22:0/1278215133 learned_addr learned my addr 172.21.15.22:0/1278215133 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:39:51.677 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.292+0000 7f7a837fe700 1 -- 172.21.15.22:0/1278215133 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a841076b0 con 0x7f7a84106550 2024-03-24T09:39:51.677 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.292+0000 7f7a837fe700 1 --2- 172.21.15.22:0/1278215133 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7a84106550 0x7f7a84106970 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7a740098a0 tx=0x7f7a74009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28f3d8499e2acf0d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:39:51.677 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.292+0000 7f7a827fc700 1 -- 172.21.15.22:0/1278215133 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a74004180 con 0x7f7a84106550 2024-03-24T09:39:51.677 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.292+0000 7f7a827fc700 1 -- 172.21.15.22:0/1278215133 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f7a740042e0 con 0x7f7a84106550 2024-03-24T09:39:51.677 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.292+0000 7f7a827fc700 1 -- 172.21.15.22:0/1278215133 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a74011a70 con 0x7f7a84106550 2024-03-24T09:39:51.678 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.292+0000 7f7a8a1ef700 1 -- 172.21.15.22:0/1278215133 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7a84106550 msgr2=0x7f7a84106970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:39:51.678 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.292+0000 7f7a8a1ef700 1 --2- 172.21.15.22:0/1278215133 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7a84106550 0x7f7a84106970 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7a740098a0 tx=0x7f7a74009bb0 comp rx=0 tx=0).stop 2024-03-24T09:39:51.678 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.292+0000 7f7a8a1ef700 1 -- 172.21.15.22:0/1278215133 shutdown_connections 2024-03-24T09:39:51.678 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.292+0000 7f7a8a1ef700 1 --2- 172.21.15.22:0/1278215133 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7a84106550 0x7f7a84106970 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:39:51.678 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.292+0000 7f7a8a1ef700 1 -- 172.21.15.22:0/1278215133 >> 172.21.15.22:0/1278215133 conn(0x7f7a84101b10 msgr2=0x7f7a84103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:39:51.678 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.293+0000 7f7a8a1ef700 1 -- 172.21.15.22:0/1278215133 shutdown_connections 2024-03-24T09:39:51.678 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.293+0000 7f7a8a1ef700 1 -- 172.21.15.22:0/1278215133 wait complete. 2024-03-24T09:39:51.678 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.293+0000 7f7a8a1ef700 1 Processor -- start 2024-03-24T09:39:51.678 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.293+0000 7f7a8a1ef700 1 -- start start 2024-03-24T09:39:51.678 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.293+0000 7f7a8a1ef700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7a84106550 0x7f7a8419a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:39:51.678 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.293+0000 7f7a8a1ef700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8419a780 con 0x7f7a84106550 2024-03-24T09:39:51.678 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.293+0000 7f7a837fe700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7a84106550 0x7f7a8419a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:39:51.678 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.294+0000 7f7a837fe700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7a84106550 0x7f7a8419a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:59656/0 (socket says 172.21.15.22:59656) 2024-03-24T09:39:51.679 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.294+0000 7f7a837fe700 1 -- 172.21.15.22:0/1649476045 learned_addr learned my addr 172.21.15.22:0/1649476045 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:39:51.679 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.294+0000 7f7a837fe700 1 -- 172.21.15.22:0/1649476045 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a74009550 con 0x7f7a84106550 2024-03-24T09:39:51.679 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.294+0000 7f7a837fe700 1 --2- 172.21.15.22:0/1649476045 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7a84106550 0x7f7a8419a240 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7a74011710 tx=0x7f7a740047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:39:51.679 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.294+0000 7f7a80ff9700 1 -- 172.21.15.22:0/1649476045 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a74003b10 con 0x7f7a84106550 2024-03-24T09:39:51.679 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.294+0000 7f7a80ff9700 1 -- 172.21.15.22:0/1649476045 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f7a7401b8c0 con 0x7f7a84106550 2024-03-24T09:39:51.679 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.294+0000 7f7a8a1ef700 1 -- 172.21.15.22:0/1649476045 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a8419a980 con 0x7f7a84106550 2024-03-24T09:39:51.679 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.294+0000 7f7a80ff9700 1 -- 172.21.15.22:0/1649476045 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a74019710 con 0x7f7a84106550 2024-03-24T09:39:51.679 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.294+0000 7f7a8a1ef700 1 -- 172.21.15.22:0/1649476045 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a8419ae20 con 0x7f7a84106550 2024-03-24T09:39:51.679 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.295+0000 7f7a80ff9700 1 -- 172.21.15.22:0/1649476045 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 4) v1 ==== 45537+0+0 (secure 0 0 0) 0x7f7a740198c0 con 0x7f7a84106550 2024-03-24T09:39:51.679 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.295+0000 7f7a8a1ef700 1 -- 172.21.15.22:0/1649476045 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a8410a8e0 con 0x7f7a84106550 2024-03-24T09:39:51.679 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.295+0000 7f7a80ff9700 1 --2- 172.21.15.22:0/1649476045 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7f7a6c0386e0 0x7f7a6c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:39:51.679 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.295+0000 7f7a80ff9700 1 -- 172.21.15.22:0/1649476045 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7a7404cc20 con 0x7f7a84106550 2024-03-24T09:39:51.679 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.295+0000 7f7a82ffd700 1 --2- 172.21.15.22:0/1649476045 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7f7a6c0386e0 0x7f7a6c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:39:51.679 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.296+0000 7f7a82ffd700 1 --2- 172.21.15.22:0/1649476045 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7f7a6c0386e0 0x7f7a6c03aba0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7a78006fd0 tx=0x7f7a78006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:39:51.680 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.300+0000 7f7a80ff9700 1 -- 172.21.15.22:0/1649476045 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7a74018b80 con 0x7f7a84106550 2024-03-24T09:39:51.680 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:50.556+0000 7f7a8a1ef700 1 -- 172.21.15.22:0/1649476045 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f7a8419da80 con 0x7f7a84106550 2024-03-24T09:39:51.680 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:51.069+0000 7f7a80ff9700 1 -- 172.21.15.22:0/1649476045 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mgrmap(e 5) v1 ==== 45548+0+0 (secure 0 0 0) 0x7f7a7401a8f0 con 0x7f7a84106550 2024-03-24T09:39:51.680 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:51.071+0000 7f7a80ff9700 1 -- 172.21.15.22:0/1649476045 <== mon.0 v2:172.21.15.22:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f7a7404c640 con 0x7f7a84106550 2024-03-24T09:39:51.680 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:51.073+0000 7f7a8a1ef700 1 -- 172.21.15.22:0/1649476045 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7f7a6c0386e0 msgr2=0x7f7a6c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:39:51.680 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:51.073+0000 7f7a8a1ef700 1 --2- 172.21.15.22:0/1649476045 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7f7a6c0386e0 0x7f7a6c03aba0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7a78006fd0 tx=0x7f7a78006e40 comp rx=0 tx=0).stop 2024-03-24T09:39:51.680 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:51.073+0000 7f7a8a1ef700 1 -- 172.21.15.22:0/1649476045 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7a84106550 msgr2=0x7f7a8419a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:39:51.680 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:51.073+0000 7f7a8a1ef700 1 --2- 172.21.15.22:0/1649476045 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7a84106550 0x7f7a8419a240 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7a74011710 tx=0x7f7a740047c0 comp rx=0 tx=0).stop 2024-03-24T09:39:51.680 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:51.073+0000 7f7a8a1ef700 1 -- 172.21.15.22:0/1649476045 shutdown_connections 2024-03-24T09:39:51.680 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:51.073+0000 7f7a8a1ef700 1 --2- 172.21.15.22:0/1649476045 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7f7a6c0386e0 0x7f7a6c03aba0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:39:51.680 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:51.073+0000 7f7a8a1ef700 1 --2- 172.21.15.22:0/1649476045 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7a84106550 0x7f7a8419a240 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:39:51.680 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:51.073+0000 7f7a8a1ef700 1 -- 172.21.15.22:0/1649476045 >> 172.21.15.22:0/1649476045 conn(0x7f7a84101b10 msgr2=0x7f7a84102670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:39:51.681 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:51.074+0000 7f7a8a1ef700 1 -- 172.21.15.22:0/1649476045 shutdown_connections 2024-03-24T09:39:51.681 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:51.074+0000 7f7a8a1ef700 1 -- 172.21.15.22:0/1649476045 wait complete. 2024-03-24T09:39:52.267 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:52 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/1649476045' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-24T09:39:52.267 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:52 smithi022 ceph-mon[29512]: mgrmap e5: smithi022.emgmxz(active, since 12s) 2024-03-24T09:39:52.950 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout { 2024-03-24T09:39:52.950 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-24T09:39:52.950 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T09:39:52.950 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "active_name": "smithi022.emgmxz", 2024-03-24T09:39:52.950 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-24T09:39:52.950 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout } 2024-03-24T09:39:52.950 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.062+0000 7f69a226d700 1 Processor -- start 2024-03-24T09:39:52.950 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.062+0000 7f69a226d700 1 -- start start 2024-03-24T09:39:52.950 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.062+0000 7f69a226d700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f699c104db0 0x7f699c1051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:39:52.950 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.062+0000 7f69a226d700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f699c079e10 con 0x7f699c104db0 2024-03-24T09:39:52.951 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.062+0000 7f699bfff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f699c104db0 0x7f699c1051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:39:52.951 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.062+0000 7f699bfff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f699c104db0 0x7f699c1051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:59674/0 (socket says 172.21.15.22:59674) 2024-03-24T09:39:52.951 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.062+0000 7f699bfff700 1 -- 172.21.15.22:0/717933505 learned_addr learned my addr 172.21.15.22:0/717933505 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:39:52.951 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.062+0000 7f699bfff700 1 -- 172.21.15.22:0/717933505 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f699c105710 con 0x7f699c104db0 2024-03-24T09:39:52.951 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.062+0000 7f699bfff700 1 --2- 172.21.15.22:0/717933505 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f699c104db0 0x7f699c1051d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f698c0098a0 tx=0x7f698c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e89646f9a1c5f155 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:39:52.951 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.063+0000 7f699affd700 1 -- 172.21.15.22:0/717933505 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f698c004180 con 0x7f699c104db0 2024-03-24T09:39:52.951 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.063+0000 7f699affd700 1 -- 172.21.15.22:0/717933505 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f698c0042e0 con 0x7f699c104db0 2024-03-24T09:39:52.951 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.063+0000 7f699affd700 1 -- 172.21.15.22:0/717933505 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f698c011a70 con 0x7f699c104db0 2024-03-24T09:39:52.951 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.063+0000 7f69a226d700 1 -- 172.21.15.22:0/717933505 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f699c104db0 msgr2=0x7f699c1051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:39:52.951 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.063+0000 7f69a226d700 1 --2- 172.21.15.22:0/717933505 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f699c104db0 0x7f699c1051d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f698c0098a0 tx=0x7f698c009bb0 comp rx=0 tx=0).stop 2024-03-24T09:39:52.951 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.063+0000 7f69a226d700 1 -- 172.21.15.22:0/717933505 shutdown_connections 2024-03-24T09:39:52.951 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.063+0000 7f69a226d700 1 --2- 172.21.15.22:0/717933505 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f699c104db0 0x7f699c1051d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:39:52.951 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.063+0000 7f69a226d700 1 -- 172.21.15.22:0/717933505 >> 172.21.15.22:0/717933505 conn(0x7f699c100950 msgr2=0x7f699c102db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:39:52.952 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.063+0000 7f69a226d700 1 -- 172.21.15.22:0/717933505 shutdown_connections 2024-03-24T09:39:52.952 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.063+0000 7f69a226d700 1 -- 172.21.15.22:0/717933505 wait complete. 2024-03-24T09:39:52.952 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.064+0000 7f69a226d700 1 Processor -- start 2024-03-24T09:39:52.952 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.064+0000 7f69a226d700 1 -- start start 2024-03-24T09:39:52.952 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.064+0000 7f69a226d700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f699c104db0 0x7f699c19c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:39:52.952 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.064+0000 7f69a226d700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f699c19c9a0 con 0x7f699c104db0 2024-03-24T09:39:52.952 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.064+0000 7f699bfff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f699c104db0 0x7f699c19c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:39:52.952 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.065+0000 7f699bfff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f699c104db0 0x7f699c19c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:59684/0 (socket says 172.21.15.22:59684) 2024-03-24T09:39:52.952 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.065+0000 7f699bfff700 1 -- 172.21.15.22:0/1250067184 learned_addr learned my addr 172.21.15.22:0/1250067184 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:39:52.952 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.065+0000 7f699bfff700 1 -- 172.21.15.22:0/1250067184 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f698c009550 con 0x7f699c104db0 2024-03-24T09:39:52.952 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.065+0000 7f699bfff700 1 --2- 172.21.15.22:0/1250067184 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f699c104db0 0x7f699c19c460 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f698c011710 tx=0x7f698c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:39:52.952 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.065+0000 7f69997fa700 1 -- 172.21.15.22:0/1250067184 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f698c003b10 con 0x7f699c104db0 2024-03-24T09:39:52.952 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.065+0000 7f69997fa700 1 -- 172.21.15.22:0/1250067184 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f698c01b8c0 con 0x7f699c104db0 2024-03-24T09:39:52.952 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.065+0000 7f69997fa700 1 -- 172.21.15.22:0/1250067184 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f698c0197e0 con 0x7f699c104db0 2024-03-24T09:39:52.953 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.065+0000 7f69a226d700 1 -- 172.21.15.22:0/1250067184 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f699c19cba0 con 0x7f699c104db0 2024-03-24T09:39:52.953 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.065+0000 7f69a226d700 1 -- 172.21.15.22:0/1250067184 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f699c19d040 con 0x7f699c104db0 2024-03-24T09:39:52.953 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.066+0000 7f69997fa700 1 -- 172.21.15.22:0/1250067184 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 45548+0+0 (secure 0 0 0) 0x7f698c019a20 con 0x7f699c104db0 2024-03-24T09:39:52.953 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.066+0000 7f69a226d700 1 -- 172.21.15.22:0/1250067184 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f699c196130 con 0x7f699c104db0 2024-03-24T09:39:52.953 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.066+0000 7f69997fa700 1 --2- 172.21.15.22:0/1250067184 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7f6984038720 0x7f698403abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:39:52.953 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.066+0000 7f69997fa700 1 -- 172.21.15.22:0/1250067184 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f698c04c9c0 con 0x7f699c104db0 2024-03-24T09:39:52.953 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.066+0000 7f699b7fe700 1 -- 172.21.15.22:0/1250067184 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7f6984038720 msgr2=0x7f698403abe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/2423771373 2024-03-24T09:39:52.953 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.066+0000 7f699b7fe700 1 --2- 172.21.15.22:0/1250067184 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7f6984038720 0x7f698403abe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T09:39:52.953 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.071+0000 7f69997fa700 1 -- 172.21.15.22:0/1250067184 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f698c018db0 con 0x7f699c104db0 2024-03-24T09:39:52.953 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.267+0000 7f699b7fe700 1 -- 172.21.15.22:0/1250067184 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7f6984038720 msgr2=0x7f698403abe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/2423771373 2024-03-24T09:39:52.953 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.267+0000 7f699b7fe700 1 --2- 172.21.15.22:0/1250067184 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7f6984038720 0x7f698403abe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T09:39:52.953 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.332+0000 7f69a226d700 1 -- 172.21.15.22:0/1250067184 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f699c02ccb0 con 0x7f699c104db0 2024-03-24T09:39:52.953 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.332+0000 7f69997fa700 1 -- 172.21.15.22:0/1250067184 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f698c01f070 con 0x7f699c104db0 2024-03-24T09:39:52.953 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.334+0000 7f69a226d700 1 -- 172.21.15.22:0/1250067184 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7f6984038720 msgr2=0x7f698403abe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T09:39:52.954 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.334+0000 7f69a226d700 1 --2- 172.21.15.22:0/1250067184 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7f6984038720 0x7f698403abe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:39:52.954 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.334+0000 7f69a226d700 1 -- 172.21.15.22:0/1250067184 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f699c104db0 msgr2=0x7f699c19c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:39:52.954 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.334+0000 7f69a226d700 1 --2- 172.21.15.22:0/1250067184 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f699c104db0 0x7f699c19c460 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f698c011710 tx=0x7f698c0047c0 comp rx=0 tx=0).stop 2024-03-24T09:39:52.954 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.334+0000 7f69a226d700 1 -- 172.21.15.22:0/1250067184 shutdown_connections 2024-03-24T09:39:52.954 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.334+0000 7f69a226d700 1 --2- 172.21.15.22:0/1250067184 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7f6984038720 0x7f698403abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:39:52.954 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.334+0000 7f69a226d700 1 --2- 172.21.15.22:0/1250067184 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f699c104db0 0x7f699c19c460 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:39:52.954 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.334+0000 7f69a226d700 1 -- 172.21.15.22:0/1250067184 >> 172.21.15.22:0/1250067184 conn(0x7f699c100950 msgr2=0x7f699c193270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:39:52.954 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.334+0000 7f69a226d700 1 -- 172.21.15.22:0/1250067184 shutdown_connections 2024-03-24T09:39:52.954 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:52.334+0000 7f69a226d700 1 -- 172.21.15.22:0/1250067184 wait complete. 2024-03-24T09:39:52.954 INFO:teuthology.orchestra.run.smithi022.stdout:Waiting for the mgr to restart... 2024-03-24T09:39:52.954 INFO:teuthology.orchestra.run.smithi022.stdout:Waiting for mgr epoch 5... 2024-03-24T09:39:53.210 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:53 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/1250067184' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-24T09:39:58.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:58 smithi022 ceph-mon[29512]: Active manager daemon smithi022.emgmxz restarted 2024-03-24T09:39:58.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:58 smithi022 ceph-mon[29512]: Activating manager daemon smithi022.emgmxz 2024-03-24T09:39:58.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:58 smithi022 ceph-mon[29512]: osdmap e2: 0 total, 0 up, 0 in 2024-03-24T09:39:58.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:58 smithi022 ceph-mon[29512]: mgrmap e6: smithi022.emgmxz(active, starting, since 0.00329196s) 2024-03-24T09:39:58.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:58 smithi022 ceph-mon[29512]: from='mgr.14120 172.21.15.22:0/3193617044' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mon metadata", "id": "smithi022"}]: dispatch 2024-03-24T09:39:58.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:58 smithi022 ceph-mon[29512]: from='mgr.14120 172.21.15.22:0/3193617044' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mgr metadata", "who": "smithi022.emgmxz", "id": "smithi022.emgmxz"}]: dispatch 2024-03-24T09:39:58.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:58 smithi022 ceph-mon[29512]: from='mgr.14120 172.21.15.22:0/3193617044' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T09:39:58.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:58 smithi022 ceph-mon[29512]: from='mgr.14120 172.21.15.22:0/3193617044' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T09:39:58.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:58 smithi022 ceph-mon[29512]: from='mgr.14120 172.21.15.22:0/3193617044' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T09:39:58.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:58 smithi022 ceph-mon[29512]: Manager daemon smithi022.emgmxz is now available 2024-03-24T09:39:58.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:58 smithi022 ceph-mon[29512]: from='mgr.14120 172.21.15.22:0/3193617044' entity='mgr.smithi022.emgmxz' 2024-03-24T09:39:58.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:58 smithi022 ceph-mon[29512]: from='mgr.14120 172.21.15.22:0/3193617044' entity='mgr.smithi022.emgmxz' 2024-03-24T09:39:58.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:58 smithi022 ceph-mon[29512]: from='mgr.14120 172.21.15.22:0/3193617044' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:39:58.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:58 smithi022 ceph-mon[29512]: from='mgr.14120 172.21.15.22:0/3193617044' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:39:58.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:58 smithi022 ceph-mon[29512]: from='mgr.14120 172.21.15.22:0/3193617044' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:39:59.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:59 smithi022 ceph-mon[29512]: Found migration_current of "None". Setting to last migration. 2024-03-24T09:39:59.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:59 smithi022 ceph-mon[29512]: from='mgr.14120 172.21.15.22:0/3193617044' entity='mgr.smithi022.emgmxz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.emgmxz/mirror_snapshot_schedule"}]: dispatch 2024-03-24T09:39:59.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:59 smithi022 ceph-mon[29512]: from='mgr.14120 172.21.15.22:0/3193617044' entity='mgr.smithi022.emgmxz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.emgmxz/trash_purge_schedule"}]: dispatch 2024-03-24T09:39:59.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:59 smithi022 ceph-mon[29512]: from='mgr.14120 172.21.15.22:0/3193617044' entity='mgr.smithi022.emgmxz' 2024-03-24T09:39:59.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:59 smithi022 ceph-mon[29512]: from='mgr.14120 172.21.15.22:0/3193617044' entity='mgr.smithi022.emgmxz' 2024-03-24T09:39:59.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:39:59 smithi022 ceph-mon[29512]: mgrmap e7: smithi022.emgmxz(active, since 1.00503s) 2024-03-24T09:39:59.591 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout { 2024-03-24T09:39:59.591 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-24T09:39:59.591 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-24T09:39:59.591 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout } 2024-03-24T09:39:59.591 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.346+0000 7fe0a306d700 1 Processor -- start 2024-03-24T09:39:59.591 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.346+0000 7fe0a306d700 1 -- start start 2024-03-24T09:39:59.591 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.346+0000 7fe0a306d700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe09c07bdc0 0x7fe09c07a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:39:59.591 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.346+0000 7fe0a306d700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe09c07a800 con 0x7fe09c07bdc0 2024-03-24T09:39:59.592 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.346+0000 7fe0a0e09700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe09c07bdc0 0x7fe09c07a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:39:59.592 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.346+0000 7fe0a0e09700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe09c07bdc0 0x7fe09c07a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:59694/0 (socket says 172.21.15.22:59694) 2024-03-24T09:39:59.592 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.346+0000 7fe0a0e09700 1 -- 172.21.15.22:0/322434660 learned_addr learned my addr 172.21.15.22:0/322434660 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:39:59.592 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.347+0000 7fe0a0e09700 1 -- 172.21.15.22:0/322434660 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe09c07a940 con 0x7fe09c07bdc0 2024-03-24T09:39:59.592 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.347+0000 7fe0a0e09700 1 --2- 172.21.15.22:0/322434660 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe09c07bdc0 0x7fe09c07a2c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe0900098a0 tx=0x7fe090009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e3ae43c5fd14fa6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:39:59.592 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.347+0000 7fe09b7fe700 1 -- 172.21.15.22:0/322434660 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe090004180 con 0x7fe09c07bdc0 2024-03-24T09:39:59.592 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.347+0000 7fe09b7fe700 1 -- 172.21.15.22:0/322434660 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fe0900042e0 con 0x7fe09c07bdc0 2024-03-24T09:39:59.592 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.347+0000 7fe09b7fe700 1 -- 172.21.15.22:0/322434660 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe090011a70 con 0x7fe09c07bdc0 2024-03-24T09:39:59.592 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.348+0000 7fe0a306d700 1 -- 172.21.15.22:0/322434660 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe09c07bdc0 msgr2=0x7fe09c07a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:39:59.592 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.348+0000 7fe0a306d700 1 --2- 172.21.15.22:0/322434660 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe09c07bdc0 0x7fe09c07a2c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe0900098a0 tx=0x7fe090009bb0 comp rx=0 tx=0).stop 2024-03-24T09:39:59.592 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.348+0000 7fe0a306d700 1 -- 172.21.15.22:0/322434660 shutdown_connections 2024-03-24T09:39:59.592 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.348+0000 7fe0a306d700 1 --2- 172.21.15.22:0/322434660 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe09c07bdc0 0x7fe09c07a2c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:39:59.592 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.348+0000 7fe0a306d700 1 -- 172.21.15.22:0/322434660 >> 172.21.15.22:0/322434660 conn(0x7fe09c101b10 msgr2=0x7fe09c103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:39:59.593 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.348+0000 7fe0a306d700 1 -- 172.21.15.22:0/322434660 shutdown_connections 2024-03-24T09:39:59.593 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.348+0000 7fe0a306d700 1 -- 172.21.15.22:0/322434660 wait complete. 2024-03-24T09:39:59.593 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.348+0000 7fe0a306d700 1 Processor -- start 2024-03-24T09:39:59.593 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.348+0000 7fe0a306d700 1 -- start start 2024-03-24T09:39:59.593 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.348+0000 7fe0a306d700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe09c07bdc0 0x7fe09c19c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:39:59.593 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.348+0000 7fe0a306d700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe09c19c8b0 con 0x7fe09c07bdc0 2024-03-24T09:39:59.593 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.349+0000 7fe0a0e09700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe09c07bdc0 0x7fe09c19c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:39:59.593 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.349+0000 7fe0a0e09700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe09c07bdc0 0x7fe09c19c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:59710/0 (socket says 172.21.15.22:59710) 2024-03-24T09:39:59.593 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.349+0000 7fe0a0e09700 1 -- 172.21.15.22:0/1336838769 learned_addr learned my addr 172.21.15.22:0/1336838769 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:39:59.593 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.349+0000 7fe0a0e09700 1 -- 172.21.15.22:0/1336838769 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe090009550 con 0x7fe09c07bdc0 2024-03-24T09:39:59.593 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.349+0000 7fe0a0e09700 1 --2- 172.21.15.22:0/1336838769 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe09c07bdc0 0x7fe09c19c370 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe090011710 tx=0x7fe0900047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:39:59.593 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.349+0000 7fe099ffb700 1 -- 172.21.15.22:0/1336838769 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe090003b10 con 0x7fe09c07bdc0 2024-03-24T09:39:59.593 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.349+0000 7fe099ffb700 1 -- 172.21.15.22:0/1336838769 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fe09001b8c0 con 0x7fe09c07bdc0 2024-03-24T09:39:59.593 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.349+0000 7fe0a306d700 1 -- 172.21.15.22:0/1336838769 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe09c19cab0 con 0x7fe09c07bdc0 2024-03-24T09:39:59.594 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.349+0000 7fe099ffb700 1 -- 172.21.15.22:0/1336838769 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0900197e0 con 0x7fe09c07bdc0 2024-03-24T09:39:59.594 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.349+0000 7fe0a306d700 1 -- 172.21.15.22:0/1336838769 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe09c19cf50 con 0x7fe09c07bdc0 2024-03-24T09:39:59.594 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.350+0000 7fe099ffb700 1 -- 172.21.15.22:0/1336838769 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 45548+0+0 (secure 0 0 0) 0x7fe090019a20 con 0x7fe09c07bdc0 2024-03-24T09:39:59.594 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.351+0000 7fe099ffb700 1 --2- 172.21.15.22:0/1336838769 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7fe08c038720 0x7fe08c03abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:39:59.594 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.351+0000 7fe099ffb700 1 -- 172.21.15.22:0/1336838769 --> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe08c03b2f0 con 0x7fe08c038720 2024-03-24T09:39:59.594 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.351+0000 7fe099ffb700 1 -- 172.21.15.22:0/1336838769 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe09004cba0 con 0x7fe09c07bdc0 2024-03-24T09:39:59.594 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.351+0000 7fe09bfff700 1 -- 172.21.15.22:0/1336838769 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7fe08c038720 msgr2=0x7fe08c03abe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/2423771373 2024-03-24T09:39:59.594 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.351+0000 7fe09bfff700 1 --2- 172.21.15.22:0/1336838769 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7fe08c038720 0x7fe08c03abe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T09:39:59.594 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.552+0000 7fe09bfff700 1 -- 172.21.15.22:0/1336838769 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7fe08c038720 msgr2=0x7fe08c03abe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/2423771373 2024-03-24T09:39:59.594 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.552+0000 7fe09bfff700 1 --2- 172.21.15.22:0/1336838769 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7fe08c038720 0x7fe08c03abe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T09:39:59.594 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.953+0000 7fe09bfff700 1 -- 172.21.15.22:0/1336838769 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7fe08c038720 msgr2=0x7fe08c03abe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/2423771373 2024-03-24T09:39:59.594 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:53.953+0000 7fe09bfff700 1 --2- 172.21.15.22:0/1336838769 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7fe08c038720 0x7fe08c03abe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T09:39:59.594 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:54.754+0000 7fe09bfff700 1 -- 172.21.15.22:0/1336838769 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7fe08c038720 msgr2=0x7fe08c03abe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/2423771373 2024-03-24T09:39:59.595 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:54.754+0000 7fe09bfff700 1 --2- 172.21.15.22:0/1336838769 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7fe08c038720 0x7fe08c03abe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T09:39:59.595 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:56.355+0000 7fe09bfff700 1 -- 172.21.15.22:0/1336838769 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7fe08c038720 msgr2=0x7fe08c03abe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/2423771373 2024-03-24T09:39:59.595 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:56.355+0000 7fe09bfff700 1 --2- 172.21.15.22:0/1336838769 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7fe08c038720 0x7fe08c03abe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T09:39:59.595 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:58.020+0000 7fe099ffb700 1 -- 172.21.15.22:0/1336838769 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mgrmap(e 6) v1 ==== 45315+0+0 (secure 0 0 0) 0x7fe09000ed00 con 0x7fe09c07bdc0 2024-03-24T09:39:59.595 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:58.020+0000 7fe099ffb700 1 -- 172.21.15.22:0/1336838769 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7fe08c038720 msgr2=0x7fe08c03abe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T09:39:59.595 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:58.020+0000 7fe099ffb700 1 --2- 172.21.15.22:0/1336838769 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7fe08c038720 0x7fe08c03abe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:39:59.595 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:59.021+0000 7fe099ffb700 1 -- 172.21.15.22:0/1336838769 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mgrmap(e 7) v1 ==== 45442+0+0 (secure 0 0 0) 0x7fe09001baa0 con 0x7fe09c07bdc0 2024-03-24T09:39:59.595 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:59.021+0000 7fe099ffb700 1 --2- 172.21.15.22:0/1336838769 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fe08c03bfb0 0x7fe08c03e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:39:59.595 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:59.021+0000 7fe099ffb700 1 -- 172.21.15.22:0/1336838769 --> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe08c03b2f0 con 0x7fe08c03bfb0 2024-03-24T09:39:59.595 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:59.021+0000 7fe09bfff700 1 --2- 172.21.15.22:0/1336838769 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fe08c03bfb0 0x7fe08c03e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:39:59.595 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:59.022+0000 7fe09bfff700 1 --2- 172.21.15.22:0/1336838769 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fe08c03bfb0 0x7fe08c03e3a0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fe088003a10 tx=0x7fe088009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:39:59.595 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:59.023+0000 7fe099ffb700 1 -- 172.21.15.22:0/1336838769 <== mgr.14120 v2:172.21.15.22:6800/3036078499 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7fe08c03b2f0 con 0x7fe08c03bfb0 2024-03-24T09:39:59.595 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:59.029+0000 7fe0a306d700 1 -- 172.21.15.22:0/1336838769 --> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fe09c10c7c0 con 0x7fe08c03bfb0 2024-03-24T09:39:59.596 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:59.029+0000 7fe099ffb700 1 -- 172.21.15.22:0/1336838769 <== mgr.14120 v2:172.21.15.22:6800/3036078499 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fe09c10c7c0 con 0x7fe08c03bfb0 2024-03-24T09:39:59.596 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:59.029+0000 7fe0a306d700 1 -- 172.21.15.22:0/1336838769 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fe08c03bfb0 msgr2=0x7fe08c03e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:39:59.596 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:59.029+0000 7fe0a306d700 1 --2- 172.21.15.22:0/1336838769 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fe08c03bfb0 0x7fe08c03e3a0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fe088003a10 tx=0x7fe088009290 comp rx=0 tx=0).stop 2024-03-24T09:39:59.596 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:59.030+0000 7fe0a306d700 1 -- 172.21.15.22:0/1336838769 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe09c07bdc0 msgr2=0x7fe09c19c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:39:59.596 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:59.030+0000 7fe0a306d700 1 --2- 172.21.15.22:0/1336838769 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe09c07bdc0 0x7fe09c19c370 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe090011710 tx=0x7fe0900047c0 comp rx=0 tx=0).stop 2024-03-24T09:39:59.596 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:59.030+0000 7fe0a306d700 1 -- 172.21.15.22:0/1336838769 shutdown_connections 2024-03-24T09:39:59.596 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:59.030+0000 7fe0a306d700 1 --2- 172.21.15.22:0/1336838769 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fe08c03bfb0 0x7fe08c03e3a0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:39:59.596 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:59.030+0000 7fe0a306d700 1 --2- 172.21.15.22:0/1336838769 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe09c07bdc0 0x7fe09c19c370 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:39:59.596 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:59.030+0000 7fe0a306d700 1 --2- 172.21.15.22:0/1336838769 >> [v2:172.21.15.22:6800/2423771373,v1:172.21.15.22:6801/2423771373] conn(0x7fe08c038720 0x7fe08c03abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:39:59.596 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:59.030+0000 7fe0a306d700 1 -- 172.21.15.22:0/1336838769 >> 172.21.15.22:0/1336838769 conn(0x7fe09c101b10 msgr2=0x7fe09c102670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:39:59.596 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:59.030+0000 7fe0a306d700 1 -- 172.21.15.22:0/1336838769 shutdown_connections 2024-03-24T09:39:59.596 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:39:59.030+0000 7fe0a306d700 1 -- 172.21.15.22:0/1336838769 wait complete. 2024-03-24T09:39:59.596 INFO:teuthology.orchestra.run.smithi022.stdout:mgr epoch 5 is available 2024-03-24T09:39:59.597 INFO:teuthology.orchestra.run.smithi022.stdout:Setting orchestrator backend to cephadm... 2024-03-24T09:40:00.249 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:00 smithi022 ceph-mon[29512]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-24T09:40:00.249 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:00 smithi022 ceph-mon[29512]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-24T09:40:00.249 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:00 smithi022 ceph-mon[29512]: from='mgr.14120 172.21.15.22:0/3193617044' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:40:00.872 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.010+0000 7fcfb23a5700 1 Processor -- start 2024-03-24T09:40:00.872 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.010+0000 7fcfb23a5700 1 -- start start 2024-03-24T09:40:00.872 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.010+0000 7fcfb23a5700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcfac07bde0 0x7fcfac07a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:00.872 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.010+0000 7fcfb23a5700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac07a820 con 0x7fcfac07bde0 2024-03-24T09:40:00.872 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.010+0000 7fcfabfff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcfac07bde0 0x7fcfac07a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:00.872 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.010+0000 7fcfabfff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcfac07bde0 0x7fcfac07a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:50952/0 (socket says 172.21.15.22:50952) 2024-03-24T09:40:00.872 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.010+0000 7fcfabfff700 1 -- 172.21.15.22:0/7014144 learned_addr learned my addr 172.21.15.22:0/7014144 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:00.873 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.011+0000 7fcfabfff700 1 -- 172.21.15.22:0/7014144 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfac07a960 con 0x7fcfac07bde0 2024-03-24T09:40:00.873 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.011+0000 7fcfabfff700 1 --2- 172.21.15.22:0/7014144 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcfac07bde0 0x7fcfac07a2e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c0098a0 tx=0x7fcf9c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcb1f1f3da433f66 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:00.873 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.011+0000 7fcfaaffd700 1 -- 172.21.15.22:0/7014144 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf9c004180 con 0x7fcfac07bde0 2024-03-24T09:40:00.873 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.011+0000 7fcfaaffd700 1 -- 172.21.15.22:0/7014144 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fcf9c0042e0 con 0x7fcfac07bde0 2024-03-24T09:40:00.873 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.011+0000 7fcfaaffd700 1 -- 172.21.15.22:0/7014144 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf9c011a70 con 0x7fcfac07bde0 2024-03-24T09:40:00.873 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.011+0000 7fcfb23a5700 1 -- 172.21.15.22:0/7014144 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcfac07bde0 msgr2=0x7fcfac07a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:00.873 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.011+0000 7fcfb23a5700 1 --2- 172.21.15.22:0/7014144 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcfac07bde0 0x7fcfac07a2e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c0098a0 tx=0x7fcf9c009bb0 comp rx=0 tx=0).stop 2024-03-24T09:40:00.873 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.012+0000 7fcfb23a5700 1 -- 172.21.15.22:0/7014144 shutdown_connections 2024-03-24T09:40:00.873 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.012+0000 7fcfb23a5700 1 --2- 172.21.15.22:0/7014144 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcfac07bde0 0x7fcfac07a2e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:00.874 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.012+0000 7fcfb23a5700 1 -- 172.21.15.22:0/7014144 >> 172.21.15.22:0/7014144 conn(0x7fcfac101b10 msgr2=0x7fcfac103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:00.874 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.012+0000 7fcfb23a5700 1 -- 172.21.15.22:0/7014144 shutdown_connections 2024-03-24T09:40:00.874 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.012+0000 7fcfb23a5700 1 -- 172.21.15.22:0/7014144 wait complete. 2024-03-24T09:40:00.874 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.012+0000 7fcfb23a5700 1 Processor -- start 2024-03-24T09:40:00.874 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.012+0000 7fcfb23a5700 1 -- start start 2024-03-24T09:40:00.874 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.013+0000 7fcfb23a5700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcfac07bde0 0x7fcfac19c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:00.874 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.013+0000 7fcfb23a5700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfac19c8f0 con 0x7fcfac07bde0 2024-03-24T09:40:00.874 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.013+0000 7fcfabfff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcfac07bde0 0x7fcfac19c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:00.874 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.013+0000 7fcfabfff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcfac07bde0 0x7fcfac19c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:50960/0 (socket says 172.21.15.22:50960) 2024-03-24T09:40:00.874 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.013+0000 7fcfabfff700 1 -- 172.21.15.22:0/7323843 learned_addr learned my addr 172.21.15.22:0/7323843 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:00.874 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.013+0000 7fcfabfff700 1 -- 172.21.15.22:0/7323843 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf9c009550 con 0x7fcfac07bde0 2024-03-24T09:40:00.874 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.013+0000 7fcfabfff700 1 --2- 172.21.15.22:0/7323843 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcfac07bde0 0x7fcfac19c3b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c011710 tx=0x7fcf9c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:00.874 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.013+0000 7fcfa97fa700 1 -- 172.21.15.22:0/7323843 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf9c003b10 con 0x7fcfac07bde0 2024-03-24T09:40:00.875 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.013+0000 7fcfa97fa700 1 -- 172.21.15.22:0/7323843 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fcf9c01b8c0 con 0x7fcfac07bde0 2024-03-24T09:40:00.875 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.013+0000 7fcfb23a5700 1 -- 172.21.15.22:0/7323843 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfac19caf0 con 0x7fcfac07bde0 2024-03-24T09:40:00.875 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.013+0000 7fcfa97fa700 1 -- 172.21.15.22:0/7323843 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf9c0197e0 con 0x7fcfac07bde0 2024-03-24T09:40:00.875 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.013+0000 7fcfb23a5700 1 -- 172.21.15.22:0/7323843 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfac19cf90 con 0x7fcfac07bde0 2024-03-24T09:40:00.875 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.014+0000 7fcfa97fa700 1 -- 172.21.15.22:0/7323843 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 7) v1 ==== 45442+0+0 (secure 0 0 0) 0x7fcf9c0199d0 con 0x7fcfac07bde0 2024-03-24T09:40:00.875 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.014+0000 7fcfb23a5700 1 -- 172.21.15.22:0/7323843 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfac10c7f0 con 0x7fcfac07bde0 2024-03-24T09:40:00.875 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.014+0000 7fcfa97fa700 1 --2- 172.21.15.22:0/7323843 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fcf94038650 0x7fcf9403ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:00.875 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.014+0000 7fcfa97fa700 1 -- 172.21.15.22:0/7323843 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fcf9c04c790 con 0x7fcfac07bde0 2024-03-24T09:40:00.875 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.014+0000 7fcfab7fe700 1 --2- 172.21.15.22:0/7323843 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fcf94038650 0x7fcf9403ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:00.875 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.015+0000 7fcfab7fe700 1 --2- 172.21.15.22:0/7323843 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fcf94038650 0x7fcf9403ab10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0006fd0 tx=0x7fcfa0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:00.875 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.019+0000 7fcfa97fa700 1 -- 172.21.15.22:0/7323843 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fcf9c018db0 con 0x7fcfac07bde0 2024-03-24T09:40:00.875 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.071+0000 7fcfa97fa700 1 -- 172.21.15.22:0/7323843 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7fcf9c02ad20 con 0x7fcfac07bde0 2024-03-24T09:40:00.875 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.238+0000 7fcfb23a5700 1 -- 172.21.15.22:0/7323843 --> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fcfac102df0 con 0x7fcf94038650 2024-03-24T09:40:00.875 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.248+0000 7fcfa97fa700 1 -- 172.21.15.22:0/7323843 <== mgr.14120 v2:172.21.15.22:6800/3036078499 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fcfac102df0 con 0x7fcf94038650 2024-03-24T09:40:00.875 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.251+0000 7fcfb23a5700 1 -- 172.21.15.22:0/7323843 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fcf94038650 msgr2=0x7fcf9403ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:00.876 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.251+0000 7fcfb23a5700 1 --2- 172.21.15.22:0/7323843 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fcf94038650 0x7fcf9403ab10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0006fd0 tx=0x7fcfa0006e40 comp rx=0 tx=0).stop 2024-03-24T09:40:00.876 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.251+0000 7fcfb23a5700 1 -- 172.21.15.22:0/7323843 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcfac07bde0 msgr2=0x7fcfac19c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:00.876 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.251+0000 7fcfb23a5700 1 --2- 172.21.15.22:0/7323843 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcfac07bde0 0x7fcfac19c3b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c011710 tx=0x7fcf9c0047c0 comp rx=0 tx=0).stop 2024-03-24T09:40:00.876 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.251+0000 7fcfb23a5700 1 -- 172.21.15.22:0/7323843 shutdown_connections 2024-03-24T09:40:00.876 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.251+0000 7fcfb23a5700 1 --2- 172.21.15.22:0/7323843 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fcf94038650 0x7fcf9403ab10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:00.876 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.251+0000 7fcfb23a5700 1 --2- 172.21.15.22:0/7323843 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcfac07bde0 0x7fcfac19c3b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:00.876 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.251+0000 7fcfb23a5700 1 -- 172.21.15.22:0/7323843 >> 172.21.15.22:0/7323843 conn(0x7fcfac101b10 msgr2=0x7fcfac1026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:00.876 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.251+0000 7fcfb23a5700 1 -- 172.21.15.22:0/7323843 shutdown_connections 2024-03-24T09:40:00.876 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:00.251+0000 7fcfb23a5700 1 -- 172.21.15.22:0/7323843 wait complete. 2024-03-24T09:40:01.206 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:01 smithi022 ceph-mon[29512]: [24/Mar/2024:09:39:59] ENGINE Bus STARTING 2024-03-24T09:40:01.207 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:01 smithi022 ceph-mon[29512]: [24/Mar/2024:09:39:59] ENGINE Serving on http://172.21.15.22:8765 2024-03-24T09:40:01.207 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:01 smithi022 ceph-mon[29512]: [24/Mar/2024:09:39:59] ENGINE Serving on https://172.21.15.22:7150 2024-03-24T09:40:01.207 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:01 smithi022 ceph-mon[29512]: [24/Mar/2024:09:39:59] ENGINE Bus STARTED 2024-03-24T09:40:01.207 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:01 smithi022 ceph-mon[29512]: mgrmap e8: smithi022.emgmxz(active, since 2s) 2024-03-24T09:40:01.207 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:01 smithi022 ceph-mon[29512]: from='mgr.14120 172.21.15.22:0/3193617044' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:01.207 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:01 smithi022 ceph-mon[29512]: from='mgr.14120 172.21.15.22:0/3193617044' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:40:02.085 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:02 smithi022 ceph-mon[29512]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:40:02.092 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-24T09:40:02.092 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.318+0000 7f2442c76700 1 Processor -- start 2024-03-24T09:40:02.092 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.318+0000 7f2442c76700 1 -- start start 2024-03-24T09:40:02.093 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.318+0000 7f2442c76700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f243c07be70 0x7f243c07a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:02.093 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.318+0000 7f2442c76700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f243c07a8f0 con 0x7f243c07be70 2024-03-24T09:40:02.093 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.319+0000 7f2440a12700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f243c07be70 0x7f243c07a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:02.093 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.319+0000 7f2440a12700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f243c07be70 0x7f243c07a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:50972/0 (socket says 172.21.15.22:50972) 2024-03-24T09:40:02.093 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.319+0000 7f2440a12700 1 -- 172.21.15.22:0/3679442434 learned_addr learned my addr 172.21.15.22:0/3679442434 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:02.093 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.319+0000 7f2440a12700 1 -- 172.21.15.22:0/3679442434 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f243c07aa30 con 0x7f243c07be70 2024-03-24T09:40:02.093 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.319+0000 7f2440a12700 1 --2- 172.21.15.22:0/3679442434 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f243c07be70 0x7f243c07a320 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f242c0098a0 tx=0x7f242c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdf8bf6fd3e10a24 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:02.093 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.319+0000 7f243b7fe700 1 -- 172.21.15.22:0/3679442434 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f242c004180 con 0x7f243c07be70 2024-03-24T09:40:02.093 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.319+0000 7f243b7fe700 1 -- 172.21.15.22:0/3679442434 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f242c0042e0 con 0x7f243c07be70 2024-03-24T09:40:02.093 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.319+0000 7f243b7fe700 1 -- 172.21.15.22:0/3679442434 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f242c011a70 con 0x7f243c07be70 2024-03-24T09:40:02.093 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.320+0000 7f2442c76700 1 -- 172.21.15.22:0/3679442434 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f243c07be70 msgr2=0x7f243c07a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:02.093 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.320+0000 7f2442c76700 1 --2- 172.21.15.22:0/3679442434 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f243c07be70 0x7f243c07a320 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f242c0098a0 tx=0x7f242c009bb0 comp rx=0 tx=0).stop 2024-03-24T09:40:02.093 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.320+0000 7f2442c76700 1 -- 172.21.15.22:0/3679442434 shutdown_connections 2024-03-24T09:40:02.093 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.320+0000 7f2442c76700 1 --2- 172.21.15.22:0/3679442434 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f243c07be70 0x7f243c07a320 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:02.094 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.320+0000 7f2442c76700 1 -- 172.21.15.22:0/3679442434 >> 172.21.15.22:0/3679442434 conn(0x7f243c101b10 msgr2=0x7f243c103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:02.094 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.320+0000 7f2442c76700 1 -- 172.21.15.22:0/3679442434 shutdown_connections 2024-03-24T09:40:02.094 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.320+0000 7f2442c76700 1 -- 172.21.15.22:0/3679442434 wait complete. 2024-03-24T09:40:02.094 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.320+0000 7f2442c76700 1 Processor -- start 2024-03-24T09:40:02.094 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.321+0000 7f2442c76700 1 -- start start 2024-03-24T09:40:02.094 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.321+0000 7f2442c76700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f243c07be70 0x7f243c19c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:02.094 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.321+0000 7f2442c76700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f243c19c960 con 0x7f243c07be70 2024-03-24T09:40:02.094 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.321+0000 7f2440a12700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f243c07be70 0x7f243c19c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:02.094 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.321+0000 7f2440a12700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f243c07be70 0x7f243c19c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:50980/0 (socket says 172.21.15.22:50980) 2024-03-24T09:40:02.094 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.321+0000 7f2440a12700 1 -- 172.21.15.22:0/3331627661 learned_addr learned my addr 172.21.15.22:0/3331627661 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:02.094 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.321+0000 7f2440a12700 1 -- 172.21.15.22:0/3331627661 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f242c009550 con 0x7f243c07be70 2024-03-24T09:40:02.094 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.321+0000 7f2440a12700 1 --2- 172.21.15.22:0/3331627661 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f243c07be70 0x7f243c19c420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f242c011710 tx=0x7f242c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:02.094 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.321+0000 7f2439ffb700 1 -- 172.21.15.22:0/3331627661 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f242c003b10 con 0x7f243c07be70 2024-03-24T09:40:02.094 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.321+0000 7f2439ffb700 1 -- 172.21.15.22:0/3331627661 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f242c01b8c0 con 0x7f243c07be70 2024-03-24T09:40:02.095 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.321+0000 7f2442c76700 1 -- 172.21.15.22:0/3331627661 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f243c19cb60 con 0x7f243c07be70 2024-03-24T09:40:02.095 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.322+0000 7f2439ffb700 1 -- 172.21.15.22:0/3331627661 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f242c0197e0 con 0x7f243c07be70 2024-03-24T09:40:02.095 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.322+0000 7f2442c76700 1 -- 172.21.15.22:0/3331627661 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f243c19d000 con 0x7f243c07be70 2024-03-24T09:40:02.095 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.322+0000 7f2439ffb700 1 -- 172.21.15.22:0/3331627661 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7f242c0199d0 con 0x7f243c07be70 2024-03-24T09:40:02.095 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.323+0000 7f2442c76700 1 -- 172.21.15.22:0/3331627661 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f243c10c830 con 0x7f243c07be70 2024-03-24T09:40:02.095 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.323+0000 7f2439ffb700 1 --2- 172.21.15.22:0/3331627661 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f2424038770 0x7f242403ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:02.095 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.323+0000 7f2439ffb700 1 -- 172.21.15.22:0/3331627661 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f242c04cb90 con 0x7f243c07be70 2024-03-24T09:40:02.095 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.323+0000 7f243bfff700 1 --2- 172.21.15.22:0/3331627661 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f2424038770 0x7f242403ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:02.095 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.323+0000 7f243bfff700 1 --2- 172.21.15.22:0/3331627661 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f2424038770 0x7f242403ac30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2430006fd0 tx=0x7f2430006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:02.095 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.328+0000 7f2439ffb700 1 -- 172.21.15.22:0/3331627661 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f242c018db0 con 0x7f243c07be70 2024-03-24T09:40:02.095 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.534+0000 7f2442c76700 1 -- 172.21.15.22:0/3331627661 --> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f243c102db0 con 0x7f2424038770 2024-03-24T09:40:02.095 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.534+0000 7f2439ffb700 1 -- 172.21.15.22:0/3331627661 <== mgr.14120 v2:172.21.15.22:6800/3036078499 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f243c102db0 con 0x7f2424038770 2024-03-24T09:40:02.095 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.536+0000 7f2442c76700 1 -- 172.21.15.22:0/3331627661 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f2424038770 msgr2=0x7f242403ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:02.095 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.536+0000 7f2442c76700 1 --2- 172.21.15.22:0/3331627661 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f2424038770 0x7f242403ac30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2430006fd0 tx=0x7f2430006e40 comp rx=0 tx=0).stop 2024-03-24T09:40:02.095 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.536+0000 7f2442c76700 1 -- 172.21.15.22:0/3331627661 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f243c07be70 msgr2=0x7f243c19c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:02.096 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.536+0000 7f2442c76700 1 --2- 172.21.15.22:0/3331627661 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f243c07be70 0x7f243c19c420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f242c011710 tx=0x7f242c0047c0 comp rx=0 tx=0).stop 2024-03-24T09:40:02.096 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.536+0000 7f2442c76700 1 -- 172.21.15.22:0/3331627661 shutdown_connections 2024-03-24T09:40:02.096 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.536+0000 7f2442c76700 1 --2- 172.21.15.22:0/3331627661 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f2424038770 0x7f242403ac30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:02.096 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.536+0000 7f2442c76700 1 --2- 172.21.15.22:0/3331627661 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f243c07be70 0x7f243c19c420 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:02.096 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.536+0000 7f2442c76700 1 -- 172.21.15.22:0/3331627661 >> 172.21.15.22:0/3331627661 conn(0x7f243c101b10 msgr2=0x7f243c1026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:02.096 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.536+0000 7f2442c76700 1 -- 172.21.15.22:0/3331627661 shutdown_connections 2024-03-24T09:40:02.096 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:01.536+0000 7f2442c76700 1 -- 172.21.15.22:0/3331627661 wait complete. 2024-03-24T09:40:02.096 INFO:teuthology.orchestra.run.smithi022.stdout:Generating ssh key... 2024-03-24T09:40:03.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:03 smithi022 ceph-mon[29512]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:40:03.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:03 smithi022 ceph-mon[29512]: from='mgr.14120 172.21.15.22:0/3193617044' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:03.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:03 smithi022 ceph-mon[29512]: from='mgr.14120 172.21.15.22:0/3193617044' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:03.557 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.475+0000 7fe036a5e700 1 Processor -- start 2024-03-24T09:40:03.557 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.475+0000 7fe036a5e700 1 -- start start 2024-03-24T09:40:03.557 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.475+0000 7fe036a5e700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe030106550 0x7fe030106970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:03.557 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.475+0000 7fe036a5e700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe030106f40 con 0x7fe030106550 2024-03-24T09:40:03.557 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.476+0000 7fe02ffff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe030106550 0x7fe030106970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:03.557 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.476+0000 7fe02ffff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe030106550 0x7fe030106970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:50984/0 (socket says 172.21.15.22:50984) 2024-03-24T09:40:03.557 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.476+0000 7fe02ffff700 1 -- 172.21.15.22:0/3436295833 learned_addr learned my addr 172.21.15.22:0/3436295833 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:03.557 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.476+0000 7fe02ffff700 1 -- 172.21.15.22:0/3436295833 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0301076b0 con 0x7fe030106550 2024-03-24T09:40:03.558 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.476+0000 7fe02ffff700 1 --2- 172.21.15.22:0/3436295833 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe030106550 0x7fe030106970 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe0240098a0 tx=0x7fe024009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63cdc8b0a7300323 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:03.558 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.476+0000 7fe02effd700 1 -- 172.21.15.22:0/3436295833 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe024004180 con 0x7fe030106550 2024-03-24T09:40:03.558 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.476+0000 7fe02effd700 1 -- 172.21.15.22:0/3436295833 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fe0240042e0 con 0x7fe030106550 2024-03-24T09:40:03.558 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.476+0000 7fe02effd700 1 -- 172.21.15.22:0/3436295833 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe024011a70 con 0x7fe030106550 2024-03-24T09:40:03.558 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.477+0000 7fe036a5e700 1 -- 172.21.15.22:0/3436295833 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe030106550 msgr2=0x7fe030106970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:03.558 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.477+0000 7fe036a5e700 1 --2- 172.21.15.22:0/3436295833 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe030106550 0x7fe030106970 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe0240098a0 tx=0x7fe024009bb0 comp rx=0 tx=0).stop 2024-03-24T09:40:03.558 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.477+0000 7fe036a5e700 1 -- 172.21.15.22:0/3436295833 shutdown_connections 2024-03-24T09:40:03.558 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.477+0000 7fe036a5e700 1 --2- 172.21.15.22:0/3436295833 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe030106550 0x7fe030106970 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:03.558 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.477+0000 7fe036a5e700 1 -- 172.21.15.22:0/3436295833 >> 172.21.15.22:0/3436295833 conn(0x7fe030101b10 msgr2=0x7fe030103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:03.558 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.477+0000 7fe036a5e700 1 -- 172.21.15.22:0/3436295833 shutdown_connections 2024-03-24T09:40:03.558 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.477+0000 7fe036a5e700 1 -- 172.21.15.22:0/3436295833 wait complete. 2024-03-24T09:40:03.558 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.477+0000 7fe036a5e700 1 Processor -- start 2024-03-24T09:40:03.558 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.478+0000 7fe036a5e700 1 -- start start 2024-03-24T09:40:03.559 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.478+0000 7fe036a5e700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe030106550 0x7fe03019a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:03.559 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.478+0000 7fe036a5e700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe03019a720 con 0x7fe030106550 2024-03-24T09:40:03.559 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.478+0000 7fe02ffff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe030106550 0x7fe03019a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:03.559 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.478+0000 7fe02ffff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe030106550 0x7fe03019a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:50990/0 (socket says 172.21.15.22:50990) 2024-03-24T09:40:03.559 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.478+0000 7fe02ffff700 1 -- 172.21.15.22:0/3784248098 learned_addr learned my addr 172.21.15.22:0/3784248098 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:03.559 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.478+0000 7fe02ffff700 1 -- 172.21.15.22:0/3784248098 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe024009550 con 0x7fe030106550 2024-03-24T09:40:03.559 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.478+0000 7fe02ffff700 1 --2- 172.21.15.22:0/3784248098 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe030106550 0x7fe03019a1e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe024011710 tx=0x7fe0240047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:03.559 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.478+0000 7fe02d7fa700 1 -- 172.21.15.22:0/3784248098 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe024003b10 con 0x7fe030106550 2024-03-24T09:40:03.559 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.479+0000 7fe02d7fa700 1 -- 172.21.15.22:0/3784248098 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fe02401b8c0 con 0x7fe030106550 2024-03-24T09:40:03.559 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.479+0000 7fe036a5e700 1 -- 172.21.15.22:0/3784248098 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe03019a920 con 0x7fe030106550 2024-03-24T09:40:03.559 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.479+0000 7fe02d7fa700 1 -- 172.21.15.22:0/3784248098 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe024019770 con 0x7fe030106550 2024-03-24T09:40:03.559 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.479+0000 7fe036a5e700 1 -- 172.21.15.22:0/3784248098 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe03019adc0 con 0x7fe030106550 2024-03-24T09:40:03.559 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.479+0000 7fe02d7fa700 1 -- 172.21.15.22:0/3784248098 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7fe0240199a0 con 0x7fe030106550 2024-03-24T09:40:03.559 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.480+0000 7fe036a5e700 1 -- 172.21.15.22:0/3784248098 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe03010a8e0 con 0x7fe030106550 2024-03-24T09:40:03.559 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.480+0000 7fe02d7fa700 1 --2- 172.21.15.22:0/3784248098 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fe018038770 0x7fe01803ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:03.560 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.480+0000 7fe02d7fa700 1 -- 172.21.15.22:0/3784248098 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fe02404c900 con 0x7fe030106550 2024-03-24T09:40:03.560 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.480+0000 7fe02f7fe700 1 --2- 172.21.15.22:0/3784248098 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fe018038770 0x7fe01803ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:03.560 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.480+0000 7fe02f7fe700 1 --2- 172.21.15.22:0/3784248098 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fe018038770 0x7fe01803ac30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe020006fd0 tx=0x7fe020006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:03.560 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.485+0000 7fe02d7fa700 1 -- 172.21.15.22:0/3784248098 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe02401a750 con 0x7fe030106550 2024-03-24T09:40:03.560 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:02.694+0000 7fe036a5e700 1 -- 172.21.15.22:0/3784248098 --> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fe030102db0 con 0x7fe018038770 2024-03-24T09:40:03.560 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.009+0000 7fe02d7fa700 1 -- 172.21.15.22:0/3784248098 <== mgr.14120 v2:172.21.15.22:6800/3036078499 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe030102db0 con 0x7fe018038770 2024-03-24T09:40:03.560 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.011+0000 7fe036a5e700 1 -- 172.21.15.22:0/3784248098 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fe018038770 msgr2=0x7fe01803ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:03.560 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.011+0000 7fe036a5e700 1 --2- 172.21.15.22:0/3784248098 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fe018038770 0x7fe01803ac30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe020006fd0 tx=0x7fe020006e40 comp rx=0 tx=0).stop 2024-03-24T09:40:03.560 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.011+0000 7fe036a5e700 1 -- 172.21.15.22:0/3784248098 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe030106550 msgr2=0x7fe03019a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:03.560 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.011+0000 7fe036a5e700 1 --2- 172.21.15.22:0/3784248098 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe030106550 0x7fe03019a1e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe024011710 tx=0x7fe0240047c0 comp rx=0 tx=0).stop 2024-03-24T09:40:03.560 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.011+0000 7fe036a5e700 1 -- 172.21.15.22:0/3784248098 shutdown_connections 2024-03-24T09:40:03.560 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.011+0000 7fe036a5e700 1 --2- 172.21.15.22:0/3784248098 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fe018038770 0x7fe01803ac30 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:03.560 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.011+0000 7fe036a5e700 1 --2- 172.21.15.22:0/3784248098 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe030106550 0x7fe03019a1e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:03.560 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.011+0000 7fe036a5e700 1 -- 172.21.15.22:0/3784248098 >> 172.21.15.22:0/3784248098 conn(0x7fe030101b10 msgr2=0x7fe0301026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:03.560 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.011+0000 7fe036a5e700 1 -- 172.21.15.22:0/3784248098 shutdown_connections 2024-03-24T09:40:03.561 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.011+0000 7fe036a5e700 1 -- 172.21.15.22:0/3784248098 wait complete. 2024-03-24T09:40:04.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:04 smithi022 ceph-mon[29512]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:40:04.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:04 smithi022 ceph-mon[29512]: Generating ssh key... 2024-03-24T09:40:04.755 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDNcYlsM3foEXSl36/mFg/fy7bIaYscKfx79D+1lQ6b5DszL+AkkZZKMsTkQZjpQiKFn4gIDjgomzNLJxwJmslxl1eZpZUjHK9rHW4QACShEQdsU8Jn3rnhw9iZv6UdVEqVw/KNtO6jSdhsJaobCp8NjwNxB5sBqnuvTmSqClvHFqcaC+/cEpGrZQnjkni+kT8FlL+Mc7JrZRtRrpF/u3z2LpHUoyOcty88EyRn5vzKu4pqAuuzo9iD6of0NNhPTuI19EpjWVemSyTBi0PogQxw15sChCebC5UBJhqddUxhZ1T6H5RjCY3vl9edvGtV9YJ+rynjqfBUsBb3B5qS+WgI95IJ3dY7h4HVhKFX6s76GZ0AvfpJYngp9aZu36p1/IsW70UjwaIE7WO+HiyUGbPxoFq1xGGRbnKSALTbyzcR33LvJltGjPTuKs9eZTMUCSD0bPw3UAMfQrZ06DcdWYNZB4OYeofArGE9n4YqIkHEfV1AQgHH9SavVAvPA4LHaM= ceph-45356448-e9c2-11ee-95cd-87774f69a715 2024-03-24T09:40:04.755 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.932+0000 7f5d8ae20700 1 Processor -- start 2024-03-24T09:40:04.755 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.932+0000 7f5d8ae20700 1 -- start start 2024-03-24T09:40:04.755 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.932+0000 7f5d8ae20700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5d84107b60 0x7f5d84107f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:04.755 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.932+0000 7f5d8ae20700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d84108550 con 0x7f5d84107b60 2024-03-24T09:40:04.756 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.932+0000 7f5d88bbc700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5d84107b60 0x7f5d84107f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:04.756 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.932+0000 7f5d88bbc700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5d84107b60 0x7f5d84107f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:51004/0 (socket says 172.21.15.22:51004) 2024-03-24T09:40:04.756 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.932+0000 7f5d88bbc700 1 -- 172.21.15.22:0/3045530892 learned_addr learned my addr 172.21.15.22:0/3045530892 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:04.756 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.932+0000 7f5d88bbc700 1 -- 172.21.15.22:0/3045530892 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d84108690 con 0x7f5d84107b60 2024-03-24T09:40:04.756 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.933+0000 7f5d88bbc700 1 --2- 172.21.15.22:0/3045530892 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5d84107b60 0x7f5d84107f80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c009e80 tx=0x7f5d7c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9a086180d087aa8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:04.756 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.933+0000 7f5d837fe700 1 -- 172.21.15.22:0/3045530892 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d7c00e070 con 0x7f5d84107b60 2024-03-24T09:40:04.756 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.933+0000 7f5d837fe700 1 -- 172.21.15.22:0/3045530892 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f5d7c00b8c0 con 0x7f5d84107b60 2024-03-24T09:40:04.756 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.933+0000 7f5d837fe700 1 -- 172.21.15.22:0/3045530892 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d7c004440 con 0x7f5d84107b60 2024-03-24T09:40:04.756 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.933+0000 7f5d8ae20700 1 -- 172.21.15.22:0/3045530892 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5d84107b60 msgr2=0x7f5d84107f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:04.756 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.933+0000 7f5d8ae20700 1 --2- 172.21.15.22:0/3045530892 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5d84107b60 0x7f5d84107f80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c009e80 tx=0x7f5d7c00b330 comp rx=0 tx=0).stop 2024-03-24T09:40:04.756 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.933+0000 7f5d8ae20700 1 -- 172.21.15.22:0/3045530892 shutdown_connections 2024-03-24T09:40:04.756 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.933+0000 7f5d8ae20700 1 --2- 172.21.15.22:0/3045530892 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5d84107b60 0x7f5d84107f80 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:04.756 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.933+0000 7f5d8ae20700 1 -- 172.21.15.22:0/3045530892 >> 172.21.15.22:0/3045530892 conn(0x7f5d841001a0 msgr2=0x7f5d841025c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:04.756 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.933+0000 7f5d8ae20700 1 -- 172.21.15.22:0/3045530892 shutdown_connections 2024-03-24T09:40:04.757 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.933+0000 7f5d8ae20700 1 -- 172.21.15.22:0/3045530892 wait complete. 2024-03-24T09:40:04.757 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.934+0000 7f5d8ae20700 1 Processor -- start 2024-03-24T09:40:04.757 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.934+0000 7f5d8ae20700 1 -- start start 2024-03-24T09:40:04.757 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.934+0000 7f5d8ae20700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5d84107b60 0x7f5d8419c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:04.757 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.934+0000 7f5d8ae20700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8419c950 con 0x7f5d84107b60 2024-03-24T09:40:04.757 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.934+0000 7f5d88bbc700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5d84107b60 0x7f5d8419c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:04.757 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.934+0000 7f5d88bbc700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5d84107b60 0x7f5d8419c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:51006/0 (socket says 172.21.15.22:51006) 2024-03-24T09:40:04.757 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.934+0000 7f5d88bbc700 1 -- 172.21.15.22:0/3420787693 learned_addr learned my addr 172.21.15.22:0/3420787693 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:04.757 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.935+0000 7f5d88bbc700 1 -- 172.21.15.22:0/3420787693 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d7c009b30 con 0x7f5d84107b60 2024-03-24T09:40:04.757 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.935+0000 7f5d88bbc700 1 --2- 172.21.15.22:0/3420787693 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5d84107b60 0x7f5d8419c410 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c003c30 tx=0x7f5d7c003d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:04.757 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.935+0000 7f5d81ffb700 1 -- 172.21.15.22:0/3420787693 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d7c00e040 con 0x7f5d84107b60 2024-03-24T09:40:04.757 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.935+0000 7f5d8ae20700 1 -- 172.21.15.22:0/3420787693 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d8419cb50 con 0x7f5d84107b60 2024-03-24T09:40:04.757 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.935+0000 7f5d81ffb700 1 -- 172.21.15.22:0/3420787693 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f5d7c00bc40 con 0x7f5d84107b60 2024-03-24T09:40:04.757 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.935+0000 7f5d81ffb700 1 -- 172.21.15.22:0/3420787693 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d7c012750 con 0x7f5d84107b60 2024-03-24T09:40:04.757 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.935+0000 7f5d8ae20700 1 -- 172.21.15.22:0/3420787693 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d8419d050 con 0x7f5d84107b60 2024-03-24T09:40:04.757 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.936+0000 7f5d81ffb700 1 -- 172.21.15.22:0/3420787693 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7f5d7c019040 con 0x7f5d84107b60 2024-03-24T09:40:04.758 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.936+0000 7f5d81ffb700 1 --2- 172.21.15.22:0/3420787693 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f5d6c0386e0 0x7f5d6c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:04.758 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.936+0000 7f5d81ffb700 1 -- 172.21.15.22:0/3420787693 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f5d7c04c300 con 0x7f5d84107b60 2024-03-24T09:40:04.758 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.936+0000 7f5d8ae20700 1 -- 172.21.15.22:0/3420787693 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d840623e0 con 0x7f5d84107b60 2024-03-24T09:40:04.758 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.936+0000 7f5d83fff700 1 --2- 172.21.15.22:0/3420787693 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f5d6c0386e0 0x7f5d6c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:04.758 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.937+0000 7f5d83fff700 1 --2- 172.21.15.22:0/3420787693 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f5d6c0386e0 0x7f5d6c03aba0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5d78006fd0 tx=0x7f5d78006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:04.758 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:03.941+0000 7f5d81ffb700 1 -- 172.21.15.22:0/3420787693 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5d7c01f080 con 0x7f5d84107b60 2024-03-24T09:40:04.758 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:04.147+0000 7f5d8ae20700 1 -- 172.21.15.22:0/3420787693 --> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f5d84101300 con 0x7f5d6c0386e0 2024-03-24T09:40:04.758 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:04.147+0000 7f5d81ffb700 1 -- 172.21.15.22:0/3420787693 <== mgr.14120 v2:172.21.15.22:6800/3036078499 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f5d84101300 con 0x7f5d6c0386e0 2024-03-24T09:40:04.758 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:04.149+0000 7f5d8ae20700 1 -- 172.21.15.22:0/3420787693 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f5d6c0386e0 msgr2=0x7f5d6c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:04.758 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:04.149+0000 7f5d8ae20700 1 --2- 172.21.15.22:0/3420787693 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f5d6c0386e0 0x7f5d6c03aba0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5d78006fd0 tx=0x7f5d78006e40 comp rx=0 tx=0).stop 2024-03-24T09:40:04.758 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:04.149+0000 7f5d8ae20700 1 -- 172.21.15.22:0/3420787693 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5d84107b60 msgr2=0x7f5d8419c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:04.758 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:04.149+0000 7f5d8ae20700 1 --2- 172.21.15.22:0/3420787693 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5d84107b60 0x7f5d8419c410 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c003c30 tx=0x7f5d7c003d10 comp rx=0 tx=0).stop 2024-03-24T09:40:04.758 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:04.149+0000 7f5d8ae20700 1 -- 172.21.15.22:0/3420787693 shutdown_connections 2024-03-24T09:40:04.758 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:04.149+0000 7f5d8ae20700 1 --2- 172.21.15.22:0/3420787693 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f5d6c0386e0 0x7f5d6c03aba0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:04.758 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:04.149+0000 7f5d8ae20700 1 --2- 172.21.15.22:0/3420787693 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5d84107b60 0x7f5d8419c410 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:04.759 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:04.149+0000 7f5d8ae20700 1 -- 172.21.15.22:0/3420787693 >> 172.21.15.22:0/3420787693 conn(0x7f5d841001a0 msgr2=0x7f5d84100bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:04.759 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:04.150+0000 7f5d8ae20700 1 -- 172.21.15.22:0/3420787693 shutdown_connections 2024-03-24T09:40:04.759 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:04.150+0000 7f5d8ae20700 1 -- 172.21.15.22:0/3420787693 wait complete. 2024-03-24T09:40:04.759 INFO:teuthology.orchestra.run.smithi022.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-24T09:40:04.759 INFO:teuthology.orchestra.run.smithi022.stdout:Adding key to root@localhost authorized_keys... 2024-03-24T09:40:04.759 INFO:teuthology.orchestra.run.smithi022.stdout:Adding host smithi022... 2024-03-24T09:40:06.235 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:06 smithi022 ceph-mon[29512]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:40:07.440 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:07 smithi022 ceph-mon[29512]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi022", "addr": "172.21.15.22", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:40:07.440 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:07 smithi022 ceph-mon[29512]: Deploying cephadm binary to smithi022 2024-03-24T09:40:08.580 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout Added host 'smithi022' with addr '172.21.15.22' 2024-03-24T09:40:08.580 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.157+0000 7f391dafc700 1 Processor -- start 2024-03-24T09:40:08.580 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.157+0000 7f391dafc700 1 -- start start 2024-03-24T09:40:08.580 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.157+0000 7f391dafc700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3918106530 0x7f3918106950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:08.580 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.157+0000 7f391dafc700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3918106f20 con 0x7f3918106530 2024-03-24T09:40:08.580 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.157+0000 7f39177fe700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3918106530 0x7f3918106950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:08.581 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.157+0000 7f39177fe700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3918106530 0x7f3918106950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:51008/0 (socket says 172.21.15.22:51008) 2024-03-24T09:40:08.581 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.157+0000 7f39177fe700 1 -- 172.21.15.22:0/2615921626 learned_addr learned my addr 172.21.15.22:0/2615921626 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:08.581 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.157+0000 7f39177fe700 1 -- 172.21.15.22:0/2615921626 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3918107690 con 0x7f3918106530 2024-03-24T09:40:08.581 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.157+0000 7f39177fe700 1 --2- 172.21.15.22:0/2615921626 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3918106530 0x7f3918106950 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f39080098a0 tx=0x7f3908009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7f871812fe88bc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:08.581 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.158+0000 7f39167fc700 1 -- 172.21.15.22:0/2615921626 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3908004180 con 0x7f3918106530 2024-03-24T09:40:08.581 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.158+0000 7f39167fc700 1 -- 172.21.15.22:0/2615921626 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f39080042e0 con 0x7f3918106530 2024-03-24T09:40:08.581 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.158+0000 7f39167fc700 1 -- 172.21.15.22:0/2615921626 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3908011a70 con 0x7f3918106530 2024-03-24T09:40:08.581 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.158+0000 7f391dafc700 1 -- 172.21.15.22:0/2615921626 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3918106530 msgr2=0x7f3918106950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:08.581 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.158+0000 7f391dafc700 1 --2- 172.21.15.22:0/2615921626 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3918106530 0x7f3918106950 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f39080098a0 tx=0x7f3908009bb0 comp rx=0 tx=0).stop 2024-03-24T09:40:08.581 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.158+0000 7f391dafc700 1 -- 172.21.15.22:0/2615921626 shutdown_connections 2024-03-24T09:40:08.581 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.158+0000 7f391dafc700 1 --2- 172.21.15.22:0/2615921626 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3918106530 0x7f3918106950 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:08.581 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.158+0000 7f391dafc700 1 -- 172.21.15.22:0/2615921626 >> 172.21.15.22:0/2615921626 conn(0x7f3918101b10 msgr2=0x7f3918103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:08.581 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.158+0000 7f391dafc700 1 -- 172.21.15.22:0/2615921626 shutdown_connections 2024-03-24T09:40:08.582 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.158+0000 7f391dafc700 1 -- 172.21.15.22:0/2615921626 wait complete. 2024-03-24T09:40:08.582 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.159+0000 7f391dafc700 1 Processor -- start 2024-03-24T09:40:08.582 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.159+0000 7f391dafc700 1 -- start start 2024-03-24T09:40:08.582 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.159+0000 7f391dafc700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3918106530 0x7f391807bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:08.582 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.159+0000 7f391dafc700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f391807c040 con 0x7f3918106530 2024-03-24T09:40:08.582 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.159+0000 7f39177fe700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3918106530 0x7f391807bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:08.582 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.160+0000 7f39177fe700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3918106530 0x7f391807bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:51014/0 (socket says 172.21.15.22:51014) 2024-03-24T09:40:08.582 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.160+0000 7f39177fe700 1 -- 172.21.15.22:0/3209697766 learned_addr learned my addr 172.21.15.22:0/3209697766 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:08.582 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.160+0000 7f39177fe700 1 -- 172.21.15.22:0/3209697766 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3908009550 con 0x7f3918106530 2024-03-24T09:40:08.582 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.160+0000 7f39177fe700 1 --2- 172.21.15.22:0/3209697766 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3918106530 0x7f391807bb00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3908003df0 tx=0x7f39080047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:08.582 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.160+0000 7f3914ff9700 1 -- 172.21.15.22:0/3209697766 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3908003a00 con 0x7f3918106530 2024-03-24T09:40:08.582 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.160+0000 7f391dafc700 1 -- 172.21.15.22:0/3209697766 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f391807a280 con 0x7f3918106530 2024-03-24T09:40:08.582 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.160+0000 7f3914ff9700 1 -- 172.21.15.22:0/3209697766 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f3908003b60 con 0x7f3918106530 2024-03-24T09:40:08.582 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.160+0000 7f3914ff9700 1 -- 172.21.15.22:0/3209697766 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3908023770 con 0x7f3918106530 2024-03-24T09:40:08.583 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.160+0000 7f391dafc700 1 -- 172.21.15.22:0/3209697766 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f391807a720 con 0x7f3918106530 2024-03-24T09:40:08.583 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.161+0000 7f3914ff9700 1 -- 172.21.15.22:0/3209697766 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7f390802c3f0 con 0x7f3918106530 2024-03-24T09:40:08.583 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.161+0000 7f3914ff9700 1 --2- 172.21.15.22:0/3209697766 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f3900038730 0x7f390003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:08.583 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.161+0000 7f391dafc700 1 -- 172.21.15.22:0/3209697766 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39180623e0 con 0x7f3918106530 2024-03-24T09:40:08.583 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.161+0000 7f3914ff9700 1 -- 172.21.15.22:0/3209697766 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f390804c950 con 0x7f3918106530 2024-03-24T09:40:08.583 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.161+0000 7f3916ffd700 1 --2- 172.21.15.22:0/3209697766 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f3900038730 0x7f390003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:08.583 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.162+0000 7f3916ffd700 1 --2- 172.21.15.22:0/3209697766 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f3900038730 0x7f390003abf0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f390c006fd0 tx=0x7f390c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:08.583 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.166+0000 7f3914ff9700 1 -- 172.21.15.22:0/3209697766 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3908018940 con 0x7f3918106530 2024-03-24T09:40:08.583 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:05.372+0000 7f391dafc700 1 -- 172.21.15.22:0/3209697766 --> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi022", "addr": "172.21.15.22", "target": ["mon-mgr", ""]}) v1 -- 0x7f391807b180 con 0x7f3900038730 2024-03-24T09:40:08.583 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:08.041+0000 7f3914ff9700 1 -- 172.21.15.22:0/3209697766 <== mgr.14120 v2:172.21.15.22:6800/3036078499 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f391807b180 con 0x7f3900038730 2024-03-24T09:40:08.583 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:08.043+0000 7f391dafc700 1 -- 172.21.15.22:0/3209697766 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f3900038730 msgr2=0x7f390003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:08.583 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:08.043+0000 7f391dafc700 1 --2- 172.21.15.22:0/3209697766 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f3900038730 0x7f390003abf0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f390c006fd0 tx=0x7f390c006e40 comp rx=0 tx=0).stop 2024-03-24T09:40:08.583 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:08.043+0000 7f391dafc700 1 -- 172.21.15.22:0/3209697766 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3918106530 msgr2=0x7f391807bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:08.583 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:08.043+0000 7f391dafc700 1 --2- 172.21.15.22:0/3209697766 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3918106530 0x7f391807bb00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3908003df0 tx=0x7f39080047c0 comp rx=0 tx=0).stop 2024-03-24T09:40:08.584 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:08.043+0000 7f391dafc700 1 -- 172.21.15.22:0/3209697766 shutdown_connections 2024-03-24T09:40:08.584 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:08.043+0000 7f391dafc700 1 --2- 172.21.15.22:0/3209697766 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f3900038730 0x7f390003abf0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:08.584 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:08.043+0000 7f391dafc700 1 --2- 172.21.15.22:0/3209697766 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3918106530 0x7f391807bb00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:08.584 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:08.043+0000 7f391dafc700 1 -- 172.21.15.22:0/3209697766 >> 172.21.15.22:0/3209697766 conn(0x7f3918101b10 msgr2=0x7f3918102710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:08.584 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:08.043+0000 7f391dafc700 1 -- 172.21.15.22:0/3209697766 shutdown_connections 2024-03-24T09:40:08.584 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:08.043+0000 7f391dafc700 1 -- 172.21.15.22:0/3209697766 wait complete. 2024-03-24T09:40:08.584 INFO:teuthology.orchestra.run.smithi022.stdout:Deploying mon service with default placement... 2024-03-24T09:40:09.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:09 smithi022 ceph-mon[29512]: from='mgr.14120 172.21.15.22:0/3193617044' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:09.285 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:09 smithi022 ceph-mon[29512]: Added host smithi022 2024-03-24T09:40:09.285 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:09 smithi022 ceph-mon[29512]: from='mgr.14120 172.21.15.22:0/3193617044' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:40:10.550 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-24T09:40:10.550 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.211+0000 7f390b4c5700 1 Processor -- start 2024-03-24T09:40:10.550 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.212+0000 7f390b4c5700 1 -- start start 2024-03-24T09:40:10.550 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.212+0000 7f390b4c5700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3904104de0 0x7f3904105200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:10.550 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.212+0000 7f390b4c5700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3904079e10 con 0x7f3904104de0 2024-03-24T09:40:10.551 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.212+0000 7f3909261700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3904104de0 0x7f3904105200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:10.551 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.212+0000 7f3909261700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3904104de0 0x7f3904105200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60422/0 (socket says 172.21.15.22:60422) 2024-03-24T09:40:10.551 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.212+0000 7f3909261700 1 -- 172.21.15.22:0/1088575059 learned_addr learned my addr 172.21.15.22:0/1088575059 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:10.551 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.212+0000 7f3909261700 1 -- 172.21.15.22:0/1088575059 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3904105740 con 0x7f3904104de0 2024-03-24T09:40:10.551 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.213+0000 7f3909261700 1 --2- 172.21.15.22:0/1088575059 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3904104de0 0x7f3904105200 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f39000098a0 tx=0x7f3900009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a057cd8b47b3933 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:10.551 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.213+0000 7f38fbfff700 1 -- 172.21.15.22:0/1088575059 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3900004180 con 0x7f3904104de0 2024-03-24T09:40:10.551 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.213+0000 7f38fbfff700 1 -- 172.21.15.22:0/1088575059 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f39000042e0 con 0x7f3904104de0 2024-03-24T09:40:10.551 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.213+0000 7f38fbfff700 1 -- 172.21.15.22:0/1088575059 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3900011a70 con 0x7f3904104de0 2024-03-24T09:40:10.551 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.213+0000 7f390b4c5700 1 -- 172.21.15.22:0/1088575059 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3904104de0 msgr2=0x7f3904105200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:10.551 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.213+0000 7f390b4c5700 1 --2- 172.21.15.22:0/1088575059 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3904104de0 0x7f3904105200 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f39000098a0 tx=0x7f3900009bb0 comp rx=0 tx=0).stop 2024-03-24T09:40:10.551 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.213+0000 7f390b4c5700 1 -- 172.21.15.22:0/1088575059 shutdown_connections 2024-03-24T09:40:10.551 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.213+0000 7f390b4c5700 1 --2- 172.21.15.22:0/1088575059 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3904104de0 0x7f3904105200 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:10.551 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.213+0000 7f390b4c5700 1 -- 172.21.15.22:0/1088575059 >> 172.21.15.22:0/1088575059 conn(0x7f3904100a00 msgr2=0x7f3904102e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:10.552 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.213+0000 7f390b4c5700 1 -- 172.21.15.22:0/1088575059 shutdown_connections 2024-03-24T09:40:10.552 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.213+0000 7f390b4c5700 1 -- 172.21.15.22:0/1088575059 wait complete. 2024-03-24T09:40:10.552 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.214+0000 7f390b4c5700 1 Processor -- start 2024-03-24T09:40:10.552 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.214+0000 7f390b4c5700 1 -- start start 2024-03-24T09:40:10.552 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.214+0000 7f390b4c5700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3904104de0 0x7f390419c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:10.552 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.214+0000 7f390b4c5700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390419c940 con 0x7f3904104de0 2024-03-24T09:40:10.552 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.214+0000 7f3909261700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3904104de0 0x7f390419c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:10.552 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.214+0000 7f3909261700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3904104de0 0x7f390419c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60436/0 (socket says 172.21.15.22:60436) 2024-03-24T09:40:10.552 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.214+0000 7f3909261700 1 -- 172.21.15.22:0/3555245887 learned_addr learned my addr 172.21.15.22:0/3555245887 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:10.552 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.215+0000 7f3909261700 1 -- 172.21.15.22:0/3555245887 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3900009550 con 0x7f3904104de0 2024-03-24T09:40:10.552 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.215+0000 7f3909261700 1 --2- 172.21.15.22:0/3555245887 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3904104de0 0x7f390419c400 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3900011710 tx=0x7f39000047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:10.552 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.215+0000 7f38fa7fc700 1 -- 172.21.15.22:0/3555245887 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3900003b10 con 0x7f3904104de0 2024-03-24T09:40:10.552 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.215+0000 7f38fa7fc700 1 -- 172.21.15.22:0/3555245887 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f390001b8c0 con 0x7f3904104de0 2024-03-24T09:40:10.552 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.215+0000 7f390b4c5700 1 -- 172.21.15.22:0/3555245887 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f390419cb40 con 0x7f3904104de0 2024-03-24T09:40:10.552 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.215+0000 7f38fa7fc700 1 -- 172.21.15.22:0/3555245887 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39000197e0 con 0x7f3904104de0 2024-03-24T09:40:10.553 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.215+0000 7f390b4c5700 1 -- 172.21.15.22:0/3555245887 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f390419cfe0 con 0x7f3904104de0 2024-03-24T09:40:10.553 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.216+0000 7f38fa7fc700 1 -- 172.21.15.22:0/3555245887 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7f390001ba30 con 0x7f3904104de0 2024-03-24T09:40:10.553 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.216+0000 7f390b4c5700 1 -- 172.21.15.22:0/3555245887 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3904196110 con 0x7f3904104de0 2024-03-24T09:40:10.553 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.216+0000 7f38fa7fc700 1 --2- 172.21.15.22:0/3555245887 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f38f0038770 0x7f38f003ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:10.553 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.216+0000 7f38fa7fc700 1 -- 172.21.15.22:0/3555245887 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f390004c8f0 con 0x7f3904104de0 2024-03-24T09:40:10.553 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.216+0000 7f3908a60700 1 --2- 172.21.15.22:0/3555245887 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f38f0038770 0x7f38f003ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:10.553 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.216+0000 7f3908a60700 1 --2- 172.21.15.22:0/3555245887 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f38f0038770 0x7f38f003ac30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f38f4006fd0 tx=0x7f38f4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:10.553 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.221+0000 7f38fa7fc700 1 -- 172.21.15.22:0/3555245887 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f390002cd20 con 0x7f3904104de0 2024-03-24T09:40:10.553 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.439+0000 7f390b4c5700 1 -- 172.21.15.22:0/3555245887 --> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f39040611f0 con 0x7f38f0038770 2024-03-24T09:40:10.553 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.443+0000 7f38fa7fc700 1 -- 172.21.15.22:0/3555245887 <== mgr.14120 v2:172.21.15.22:6800/3036078499 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f39040611f0 con 0x7f38f0038770 2024-03-24T09:40:10.553 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.445+0000 7f390b4c5700 1 -- 172.21.15.22:0/3555245887 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f38f0038770 msgr2=0x7f38f003ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:10.553 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.445+0000 7f390b4c5700 1 --2- 172.21.15.22:0/3555245887 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f38f0038770 0x7f38f003ac30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f38f4006fd0 tx=0x7f38f4006e40 comp rx=0 tx=0).stop 2024-03-24T09:40:10.553 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.445+0000 7f390b4c5700 1 -- 172.21.15.22:0/3555245887 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3904104de0 msgr2=0x7f390419c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:10.553 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.445+0000 7f390b4c5700 1 --2- 172.21.15.22:0/3555245887 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3904104de0 0x7f390419c400 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3900011710 tx=0x7f39000047c0 comp rx=0 tx=0).stop 2024-03-24T09:40:10.554 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.445+0000 7f390b4c5700 1 -- 172.21.15.22:0/3555245887 shutdown_connections 2024-03-24T09:40:10.554 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.445+0000 7f390b4c5700 1 --2- 172.21.15.22:0/3555245887 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f38f0038770 0x7f38f003ac30 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:10.554 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.445+0000 7f390b4c5700 1 --2- 172.21.15.22:0/3555245887 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3904104de0 0x7f390419c400 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:10.554 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.445+0000 7f390b4c5700 1 -- 172.21.15.22:0/3555245887 >> 172.21.15.22:0/3555245887 conn(0x7f3904100a00 msgr2=0x7f3904193250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:10.554 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.446+0000 7f390b4c5700 1 -- 172.21.15.22:0/3555245887 shutdown_connections 2024-03-24T09:40:10.554 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:09.446+0000 7f390b4c5700 1 -- 172.21.15.22:0/3555245887 wait complete. 2024-03-24T09:40:10.554 INFO:teuthology.orchestra.run.smithi022.stdout:Deploying mgr service with default placement... 2024-03-24T09:40:10.668 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:10 smithi022 ceph-mon[29512]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:40:10.668 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:10 smithi022 ceph-mon[29512]: Saving service mon spec with placement count:5 2024-03-24T09:40:10.668 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:10 smithi022 ceph-mon[29512]: from='mgr.14120 172.21.15.22:0/3193617044' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:10.668 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:10 smithi022 ceph-mon[29512]: from='mgr.14120 172.21.15.22:0/3193617044' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:12.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:11 smithi022 ceph-mon[29512]: from='mgr.14120 172.21.15.22:0/3193617044' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:12.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:11 smithi022 ceph-mon[29512]: from='mgr.14120 172.21.15.22:0/3193617044' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:12.343 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-24T09:40:12.343 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.001+0000 7fc8aeaf2700 1 Processor -- start 2024-03-24T09:40:12.343 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.001+0000 7fc8aeaf2700 1 -- start start 2024-03-24T09:40:12.343 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.001+0000 7fc8aeaf2700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc8a8106f80 0x7fc8a8109370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:12.343 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.001+0000 7fc8aeaf2700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a8079e50 con 0x7fc8a8106f80 2024-03-24T09:40:12.343 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.001+0000 7fc8ac88e700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc8a8106f80 0x7fc8a8109370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:12.343 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.001+0000 7fc8ac88e700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc8a8106f80 0x7fc8a8109370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60442/0 (socket says 172.21.15.22:60442) 2024-03-24T09:40:12.343 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.001+0000 7fc8ac88e700 1 -- 172.21.15.22:0/1761763988 learned_addr learned my addr 172.21.15.22:0/1761763988 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:12.343 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.001+0000 7fc8ac88e700 1 -- 172.21.15.22:0/1761763988 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a81098b0 con 0x7fc8a8106f80 2024-03-24T09:40:12.344 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.001+0000 7fc8ac88e700 1 --2- 172.21.15.22:0/1761763988 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc8a8106f80 0x7fc8a8109370 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc89c0098a0 tx=0x7fc89c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e81a595193564646 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:12.344 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.002+0000 7fc8a77fe700 1 -- 172.21.15.22:0/1761763988 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc89c004180 con 0x7fc8a8106f80 2024-03-24T09:40:12.344 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.002+0000 7fc8a77fe700 1 -- 172.21.15.22:0/1761763988 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc89c0042e0 con 0x7fc8a8106f80 2024-03-24T09:40:12.344 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.002+0000 7fc8a77fe700 1 -- 172.21.15.22:0/1761763988 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc89c011b10 con 0x7fc8a8106f80 2024-03-24T09:40:12.344 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.002+0000 7fc8aeaf2700 1 -- 172.21.15.22:0/1761763988 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc8a8106f80 msgr2=0x7fc8a8109370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:12.344 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.002+0000 7fc8aeaf2700 1 --2- 172.21.15.22:0/1761763988 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc8a8106f80 0x7fc8a8109370 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc89c0098a0 tx=0x7fc89c009bb0 comp rx=0 tx=0).stop 2024-03-24T09:40:12.344 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.003+0000 7fc8aeaf2700 1 -- 172.21.15.22:0/1761763988 shutdown_connections 2024-03-24T09:40:12.344 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.003+0000 7fc8aeaf2700 1 --2- 172.21.15.22:0/1761763988 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc8a8106f80 0x7fc8a8109370 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:12.344 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.003+0000 7fc8aeaf2700 1 -- 172.21.15.22:0/1761763988 >> 172.21.15.22:0/1761763988 conn(0x7fc8a8100930 msgr2=0x7fc8a8102db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:12.344 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.003+0000 7fc8aeaf2700 1 -- 172.21.15.22:0/1761763988 shutdown_connections 2024-03-24T09:40:12.344 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.003+0000 7fc8aeaf2700 1 -- 172.21.15.22:0/1761763988 wait complete. 2024-03-24T09:40:12.344 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.003+0000 7fc8aeaf2700 1 Processor -- start 2024-03-24T09:40:12.344 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.004+0000 7fc8aeaf2700 1 -- start start 2024-03-24T09:40:12.344 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.004+0000 7fc8aeaf2700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc8a8106f80 0x7fc8a81067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:12.345 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.004+0000 7fc8aeaf2700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a8106cf0 con 0x7fc8a8106f80 2024-03-24T09:40:12.345 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.004+0000 7fc8ac88e700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc8a8106f80 0x7fc8a81067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:12.345 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.004+0000 7fc8ac88e700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc8a8106f80 0x7fc8a81067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60456/0 (socket says 172.21.15.22:60456) 2024-03-24T09:40:12.345 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.004+0000 7fc8ac88e700 1 -- 172.21.15.22:0/1111388698 learned_addr learned my addr 172.21.15.22:0/1111388698 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:12.345 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.004+0000 7fc8ac88e700 1 -- 172.21.15.22:0/1111388698 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc89c009550 con 0x7fc8a8106f80 2024-03-24T09:40:12.345 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.004+0000 7fc8ac88e700 1 --2- 172.21.15.22:0/1111388698 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc8a8106f80 0x7fc8a81067b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc89c0117b0 tx=0x7fc89c011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:12.345 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.005+0000 7fc8a5ffb700 1 -- 172.21.15.22:0/1111388698 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc89c0038b0 con 0x7fc8a8106f80 2024-03-24T09:40:12.345 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.005+0000 7fc8aeaf2700 1 -- 172.21.15.22:0/1111388698 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8a8104f30 con 0x7fc8a8106f80 2024-03-24T09:40:12.345 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.005+0000 7fc8a5ffb700 1 -- 172.21.15.22:0/1111388698 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc89c003a10 con 0x7fc8a8106f80 2024-03-24T09:40:12.345 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.005+0000 7fc8a5ffb700 1 -- 172.21.15.22:0/1111388698 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc89c024780 con 0x7fc8a8106f80 2024-03-24T09:40:12.345 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.005+0000 7fc8aeaf2700 1 -- 172.21.15.22:0/1111388698 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8a81053d0 con 0x7fc8a8106f80 2024-03-24T09:40:12.345 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.006+0000 7fc8aeaf2700 1 -- 172.21.15.22:0/1111388698 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8a80623e0 con 0x7fc8a8106f80 2024-03-24T09:40:12.345 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.006+0000 7fc8a5ffb700 1 -- 172.21.15.22:0/1111388698 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7fc89c0248e0 con 0x7fc8a8106f80 2024-03-24T09:40:12.345 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.006+0000 7fc8a5ffb700 1 --2- 172.21.15.22:0/1111388698 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fc898038720 0x7fc89803abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:12.346 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.006+0000 7fc8a5ffb700 1 -- 172.21.15.22:0/1111388698 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc89c04c790 con 0x7fc8a8106f80 2024-03-24T09:40:12.346 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.006+0000 7fc8a7fff700 1 --2- 172.21.15.22:0/1111388698 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fc898038720 0x7fc89803abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:12.346 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.007+0000 7fc8a7fff700 1 --2- 172.21.15.22:0/1111388698 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fc898038720 0x7fc89803abe0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc894006fd0 tx=0x7fc894006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:12.346 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.011+0000 7fc8a5ffb700 1 -- 172.21.15.22:0/1111388698 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc89c024b90 con 0x7fc8a8106f80 2024-03-24T09:40:12.346 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.242+0000 7fc8aeaf2700 1 -- 172.21.15.22:0/1111388698 --> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fc8a8101bc0 con 0x7fc898038720 2024-03-24T09:40:12.346 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.246+0000 7fc8a5ffb700 1 -- 172.21.15.22:0/1111388698 <== mgr.14120 v2:172.21.15.22:6800/3036078499 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fc8a8101bc0 con 0x7fc898038720 2024-03-24T09:40:12.346 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.248+0000 7fc8aeaf2700 1 -- 172.21.15.22:0/1111388698 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fc898038720 msgr2=0x7fc89803abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:12.346 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.248+0000 7fc8aeaf2700 1 --2- 172.21.15.22:0/1111388698 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fc898038720 0x7fc89803abe0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc894006fd0 tx=0x7fc894006e40 comp rx=0 tx=0).stop 2024-03-24T09:40:12.346 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.248+0000 7fc8aeaf2700 1 -- 172.21.15.22:0/1111388698 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc8a8106f80 msgr2=0x7fc8a81067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:12.346 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.248+0000 7fc8aeaf2700 1 --2- 172.21.15.22:0/1111388698 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc8a8106f80 0x7fc8a81067b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc89c0117b0 tx=0x7fc89c011890 comp rx=0 tx=0).stop 2024-03-24T09:40:12.346 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.248+0000 7fc8aeaf2700 1 -- 172.21.15.22:0/1111388698 shutdown_connections 2024-03-24T09:40:12.346 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.248+0000 7fc8aeaf2700 1 --2- 172.21.15.22:0/1111388698 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fc898038720 0x7fc89803abe0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:12.347 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.248+0000 7fc8aeaf2700 1 --2- 172.21.15.22:0/1111388698 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc8a8106f80 0x7fc8a81067b0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:12.347 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.248+0000 7fc8aeaf2700 1 -- 172.21.15.22:0/1111388698 >> 172.21.15.22:0/1111388698 conn(0x7fc8a8100930 msgr2=0x7fc8a81014b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:12.347 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.248+0000 7fc8aeaf2700 1 -- 172.21.15.22:0/1111388698 shutdown_connections 2024-03-24T09:40:12.347 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:11.248+0000 7fc8aeaf2700 1 -- 172.21.15.22:0/1111388698 wait complete. 2024-03-24T09:40:12.347 INFO:teuthology.orchestra.run.smithi022.stdout:Deploying crash service with default placement... 2024-03-24T09:40:12.994 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:12 smithi022 ceph-mon[29512]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:40:12.994 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:12 smithi022 ceph-mon[29512]: Saving service mgr spec with placement count:2 2024-03-24T09:40:12.994 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:12 smithi022 ceph-mon[29512]: from='mgr.14120 172.21.15.22:0/3193617044' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:13.540 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-03-24T09:40:13.540 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.759+0000 7fa3718ec700 1 Processor -- start 2024-03-24T09:40:13.540 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.759+0000 7fa3718ec700 1 -- start start 2024-03-24T09:40:13.540 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.759+0000 7fa3718ec700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa36c106f90 0x7fa36c109380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:13.540 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.759+0000 7fa3718ec700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36c079e30 con 0x7fa36c106f90 2024-03-24T09:40:13.541 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.759+0000 7fa36affd700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa36c106f90 0x7fa36c109380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:13.541 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.759+0000 7fa36affd700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa36c106f90 0x7fa36c109380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60466/0 (socket says 172.21.15.22:60466) 2024-03-24T09:40:13.541 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.759+0000 7fa36affd700 1 -- 172.21.15.22:0/1039690927 learned_addr learned my addr 172.21.15.22:0/1039690927 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:13.541 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.760+0000 7fa36affd700 1 -- 172.21.15.22:0/1039690927 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa36c1098c0 con 0x7fa36c106f90 2024-03-24T09:40:13.541 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.760+0000 7fa36affd700 1 --2- 172.21.15.22:0/1039690927 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa36c106f90 0x7fa36c109380 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa35c0098a0 tx=0x7fa35c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b97fcfc8864874ec server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:13.541 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.760+0000 7fa369ffb700 1 -- 172.21.15.22:0/1039690927 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa35c004180 con 0x7fa36c106f90 2024-03-24T09:40:13.541 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.760+0000 7fa369ffb700 1 -- 172.21.15.22:0/1039690927 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa35c0042e0 con 0x7fa36c106f90 2024-03-24T09:40:13.541 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.760+0000 7fa369ffb700 1 -- 172.21.15.22:0/1039690927 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa35c011b10 con 0x7fa36c106f90 2024-03-24T09:40:13.541 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.761+0000 7fa3718ec700 1 -- 172.21.15.22:0/1039690927 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa36c106f90 msgr2=0x7fa36c109380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:13.541 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.761+0000 7fa3718ec700 1 --2- 172.21.15.22:0/1039690927 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa36c106f90 0x7fa36c109380 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa35c0098a0 tx=0x7fa35c009bb0 comp rx=0 tx=0).stop 2024-03-24T09:40:13.541 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.761+0000 7fa3718ec700 1 -- 172.21.15.22:0/1039690927 shutdown_connections 2024-03-24T09:40:13.541 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.761+0000 7fa3718ec700 1 --2- 172.21.15.22:0/1039690927 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa36c106f90 0x7fa36c109380 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:13.541 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.761+0000 7fa3718ec700 1 -- 172.21.15.22:0/1039690927 >> 172.21.15.22:0/1039690927 conn(0x7fa36c100980 msgr2=0x7fa36c102da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:13.541 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.761+0000 7fa3718ec700 1 -- 172.21.15.22:0/1039690927 shutdown_connections 2024-03-24T09:40:13.541 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.761+0000 7fa3718ec700 1 -- 172.21.15.22:0/1039690927 wait complete. 2024-03-24T09:40:13.542 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.761+0000 7fa3718ec700 1 Processor -- start 2024-03-24T09:40:13.542 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.762+0000 7fa3718ec700 1 -- start start 2024-03-24T09:40:13.542 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.762+0000 7fa3718ec700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa36c106f90 0x7fa36c1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:13.542 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.762+0000 7fa3718ec700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36c106d00 con 0x7fa36c106f90 2024-03-24T09:40:13.542 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.762+0000 7fa36affd700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa36c106f90 0x7fa36c1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:13.542 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.762+0000 7fa36affd700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa36c106f90 0x7fa36c1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60474/0 (socket says 172.21.15.22:60474) 2024-03-24T09:40:13.542 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.762+0000 7fa36affd700 1 -- 172.21.15.22:0/2889344416 learned_addr learned my addr 172.21.15.22:0/2889344416 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:13.542 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.762+0000 7fa36affd700 1 -- 172.21.15.22:0/2889344416 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa35c009550 con 0x7fa36c106f90 2024-03-24T09:40:13.542 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.762+0000 7fa36affd700 1 --2- 172.21.15.22:0/2889344416 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa36c106f90 0x7fa36c1067c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa35c009fb0 tx=0x7fa35c011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:13.542 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.762+0000 7fa3708ea700 1 -- 172.21.15.22:0/2889344416 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa35c0038b0 con 0x7fa36c106f90 2024-03-24T09:40:13.542 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.762+0000 7fa3708ea700 1 -- 172.21.15.22:0/2889344416 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa35c003a10 con 0x7fa36c106f90 2024-03-24T09:40:13.542 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.762+0000 7fa3708ea700 1 -- 172.21.15.22:0/2889344416 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa35c019750 con 0x7fa36c106f90 2024-03-24T09:40:13.542 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.762+0000 7fa3718ec700 1 -- 172.21.15.22:0/2889344416 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa36c104f40 con 0x7fa36c106f90 2024-03-24T09:40:13.542 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.762+0000 7fa3718ec700 1 -- 172.21.15.22:0/2889344416 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa36c1053e0 con 0x7fa36c106f90 2024-03-24T09:40:13.542 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.763+0000 7fa3708ea700 1 -- 172.21.15.22:0/2889344416 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7fa35c0199c0 con 0x7fa36c106f90 2024-03-24T09:40:13.543 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.763+0000 7fa3708ea700 1 --2- 172.21.15.22:0/2889344416 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fa3540386e0 0x7fa35403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:13.543 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.763+0000 7fa3708ea700 1 -- 172.21.15.22:0/2889344416 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa35c04d9a0 con 0x7fa36c106f90 2024-03-24T09:40:13.543 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.764+0000 7fa36a7fc700 1 --2- 172.21.15.22:0/2889344416 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fa3540386e0 0x7fa35403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:13.543 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.764+0000 7fa3718ec700 1 -- 172.21.15.22:0/2889344416 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa36c0623e0 con 0x7fa36c106f90 2024-03-24T09:40:13.543 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.764+0000 7fa36a7fc700 1 --2- 172.21.15.22:0/2889344416 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fa3540386e0 0x7fa35403aba0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa360006fd0 tx=0x7fa360006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:13.543 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.769+0000 7fa3708ea700 1 -- 172.21.15.22:0/2889344416 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa35c019cc0 con 0x7fa36c106f90 2024-03-24T09:40:13.543 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.993+0000 7fa3718ec700 1 -- 172.21.15.22:0/2889344416 --> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7fa36c105e30 con 0x7fa3540386e0 2024-03-24T09:40:13.543 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:12.998+0000 7fa3708ea700 1 -- 172.21.15.22:0/2889344416 <== mgr.14120 v2:172.21.15.22:6800/3036078499 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7fa36c105e30 con 0x7fa3540386e0 2024-03-24T09:40:13.543 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.000+0000 7fa3718ec700 1 -- 172.21.15.22:0/2889344416 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fa3540386e0 msgr2=0x7fa35403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:13.543 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.000+0000 7fa3718ec700 1 --2- 172.21.15.22:0/2889344416 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fa3540386e0 0x7fa35403aba0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa360006fd0 tx=0x7fa360006e40 comp rx=0 tx=0).stop 2024-03-24T09:40:13.543 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.000+0000 7fa3718ec700 1 -- 172.21.15.22:0/2889344416 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa36c106f90 msgr2=0x7fa36c1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:13.543 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.000+0000 7fa3718ec700 1 --2- 172.21.15.22:0/2889344416 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa36c106f90 0x7fa36c1067c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa35c009fb0 tx=0x7fa35c011890 comp rx=0 tx=0).stop 2024-03-24T09:40:13.543 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.000+0000 7fa3718ec700 1 -- 172.21.15.22:0/2889344416 shutdown_connections 2024-03-24T09:40:13.543 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.000+0000 7fa3718ec700 1 --2- 172.21.15.22:0/2889344416 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fa3540386e0 0x7fa35403aba0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:13.543 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.000+0000 7fa3718ec700 1 --2- 172.21.15.22:0/2889344416 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa36c106f90 0x7fa36c1067c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:13.544 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.000+0000 7fa3718ec700 1 -- 172.21.15.22:0/2889344416 >> 172.21.15.22:0/2889344416 conn(0x7fa36c100980 msgr2=0x7fa36c1014c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:13.544 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.001+0000 7fa3718ec700 1 -- 172.21.15.22:0/2889344416 shutdown_connections 2024-03-24T09:40:13.544 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.001+0000 7fa3718ec700 1 -- 172.21.15.22:0/2889344416 wait complete. 2024-03-24T09:40:13.544 INFO:teuthology.orchestra.run.smithi022.stdout:Deploying ceph-exporter service with default placement... 2024-03-24T09:40:14.166 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:13 smithi022 ceph-mon[29512]: from='mgr.14120 172.21.15.22:0/3193617044' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:14.167 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:13 smithi022 ceph-mon[29512]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:40:14.167 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:13 smithi022 ceph-mon[29512]: Saving service crash spec with placement * 2024-03-24T09:40:14.167 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:13 smithi022 ceph-mon[29512]: from='mgr.14120 172.21.15.22:0/3193617044' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:14.791 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout Scheduled ceph-exporter update... 2024-03-24T09:40:14.791 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.911+0000 7fbdbea92700 1 Processor -- start 2024-03-24T09:40:14.791 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.911+0000 7fbdbea92700 1 -- start start 2024-03-24T09:40:14.791 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.911+0000 7fbdbea92700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbdb8104d20 0x7fbdb8107150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:14.792 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.911+0000 7fbdbea92700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb80745f0 con 0x7fbdb8104d20 2024-03-24T09:40:14.792 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.911+0000 7fbdbc82e700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbdb8104d20 0x7fbdb8107150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:14.792 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.911+0000 7fbdbc82e700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbdb8104d20 0x7fbdb8107150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60482/0 (socket says 172.21.15.22:60482) 2024-03-24T09:40:14.792 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.911+0000 7fbdbc82e700 1 -- 172.21.15.22:0/3181876501 learned_addr learned my addr 172.21.15.22:0/3181876501 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:14.792 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.912+0000 7fbdbc82e700 1 -- 172.21.15.22:0/3181876501 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb8107690 con 0x7fbdb8104d20 2024-03-24T09:40:14.792 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.912+0000 7fbdbc82e700 1 --2- 172.21.15.22:0/3181876501 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbdb8104d20 0x7fbdb8107150 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbdac0098a0 tx=0x7fbdac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a63a2fedd35b1ad9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:14.792 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.914+0000 7fbdb77fe700 1 -- 172.21.15.22:0/3181876501 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbdac004180 con 0x7fbdb8104d20 2024-03-24T09:40:14.792 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.914+0000 7fbdb77fe700 1 -- 172.21.15.22:0/3181876501 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbdac0042e0 con 0x7fbdb8104d20 2024-03-24T09:40:14.792 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.914+0000 7fbdb77fe700 1 -- 172.21.15.22:0/3181876501 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbdac011b10 con 0x7fbdb8104d20 2024-03-24T09:40:14.792 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.915+0000 7fbdbea92700 1 -- 172.21.15.22:0/3181876501 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbdb8104d20 msgr2=0x7fbdb8107150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:14.792 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.915+0000 7fbdbea92700 1 --2- 172.21.15.22:0/3181876501 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbdb8104d20 0x7fbdb8107150 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbdac0098a0 tx=0x7fbdac009bb0 comp rx=0 tx=0).stop 2024-03-24T09:40:14.793 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.915+0000 7fbdbea92700 1 -- 172.21.15.22:0/3181876501 shutdown_connections 2024-03-24T09:40:14.793 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.915+0000 7fbdbea92700 1 --2- 172.21.15.22:0/3181876501 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbdb8104d20 0x7fbdb8107150 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:14.793 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.915+0000 7fbdbea92700 1 -- 172.21.15.22:0/3181876501 >> 172.21.15.22:0/3181876501 conn(0x7fbdb81009c0 msgr2=0x7fbdb8102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:14.793 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.915+0000 7fbdbea92700 1 -- 172.21.15.22:0/3181876501 shutdown_connections 2024-03-24T09:40:14.793 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.915+0000 7fbdbea92700 1 -- 172.21.15.22:0/3181876501 wait complete. 2024-03-24T09:40:14.793 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.916+0000 7fbdbea92700 1 Processor -- start 2024-03-24T09:40:14.793 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.916+0000 7fbdbea92700 1 -- start start 2024-03-24T09:40:14.793 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.916+0000 7fbdbea92700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbdb8104d20 0x7fbdb81a0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:14.793 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.916+0000 7fbdbea92700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb81a0d70 con 0x7fbdb8104d20 2024-03-24T09:40:14.793 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.916+0000 7fbdbc82e700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbdb8104d20 0x7fbdb81a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:14.793 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.916+0000 7fbdbc82e700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbdb8104d20 0x7fbdb81a0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60496/0 (socket says 172.21.15.22:60496) 2024-03-24T09:40:14.793 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.916+0000 7fbdbc82e700 1 -- 172.21.15.22:0/835006363 learned_addr learned my addr 172.21.15.22:0/835006363 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:14.793 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.916+0000 7fbdbc82e700 1 -- 172.21.15.22:0/835006363 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdac009550 con 0x7fbdb8104d20 2024-03-24T09:40:14.794 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.917+0000 7fbdbc82e700 1 --2- 172.21.15.22:0/835006363 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbdb8104d20 0x7fbdb81a0830 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbdac009fb0 tx=0x7fbdac011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:14.794 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.917+0000 7fbdb5ffb700 1 -- 172.21.15.22:0/835006363 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbdac0038b0 con 0x7fbdb8104d20 2024-03-24T09:40:14.794 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.917+0000 7fbdb5ffb700 1 -- 172.21.15.22:0/835006363 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbdac003a10 con 0x7fbdb8104d20 2024-03-24T09:40:14.794 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.917+0000 7fbdbea92700 1 -- 172.21.15.22:0/835006363 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdb81a0f70 con 0x7fbdb8104d20 2024-03-24T09:40:14.794 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.917+0000 7fbdb5ffb700 1 -- 172.21.15.22:0/835006363 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbdac019750 con 0x7fbdb8104d20 2024-03-24T09:40:14.794 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.917+0000 7fbdbea92700 1 -- 172.21.15.22:0/835006363 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdb81a1410 con 0x7fbdb8104d20 2024-03-24T09:40:14.794 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.918+0000 7fbdb5ffb700 1 -- 172.21.15.22:0/835006363 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7fbdac0198b0 con 0x7fbdb8104d20 2024-03-24T09:40:14.794 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.918+0000 7fbdb5ffb700 1 --2- 172.21.15.22:0/835006363 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fbda8038730 0x7fbda803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:14.794 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.918+0000 7fbdb5ffb700 1 -- 172.21.15.22:0/835006363 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fbdac04d890 con 0x7fbdb8104d20 2024-03-24T09:40:14.794 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.918+0000 7fbdbea92700 1 -- 172.21.15.22:0/835006363 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdb819a000 con 0x7fbdb8104d20 2024-03-24T09:40:14.794 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.918+0000 7fbdb7fff700 1 --2- 172.21.15.22:0/835006363 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fbda8038730 0x7fbda803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:14.794 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.919+0000 7fbdb7fff700 1 --2- 172.21.15.22:0/835006363 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fbda8038730 0x7fbda803abf0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fbda4006fd0 tx=0x7fbda4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:14.795 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:13.923+0000 7fbdb5ffb700 1 -- 172.21.15.22:0/835006363 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbdac048020 con 0x7fbdb8104d20 2024-03-24T09:40:14.795 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:14.150+0000 7fbdbea92700 1 -- 172.21.15.22:0/835006363 --> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7fbdb80611f0 con 0x7fbda8038730 2024-03-24T09:40:14.795 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:14.154+0000 7fbdb5ffb700 1 -- 172.21.15.22:0/835006363 <== mgr.14120 v2:172.21.15.22:6800/3036078499 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7fbdb80611f0 con 0x7fbda8038730 2024-03-24T09:40:14.795 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:14.156+0000 7fbdbea92700 1 -- 172.21.15.22:0/835006363 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fbda8038730 msgr2=0x7fbda803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:14.795 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:14.156+0000 7fbdbea92700 1 --2- 172.21.15.22:0/835006363 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fbda8038730 0x7fbda803abf0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fbda4006fd0 tx=0x7fbda4006e40 comp rx=0 tx=0).stop 2024-03-24T09:40:14.795 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:14.156+0000 7fbdbea92700 1 -- 172.21.15.22:0/835006363 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbdb8104d20 msgr2=0x7fbdb81a0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:14.795 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:14.156+0000 7fbdbea92700 1 --2- 172.21.15.22:0/835006363 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbdb8104d20 0x7fbdb81a0830 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbdac009fb0 tx=0x7fbdac011890 comp rx=0 tx=0).stop 2024-03-24T09:40:14.795 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:14.156+0000 7fbdbea92700 1 -- 172.21.15.22:0/835006363 shutdown_connections 2024-03-24T09:40:14.795 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:14.156+0000 7fbdbea92700 1 --2- 172.21.15.22:0/835006363 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fbda8038730 0x7fbda803abf0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:14.795 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:14.156+0000 7fbdbea92700 1 --2- 172.21.15.22:0/835006363 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbdb8104d20 0x7fbdb81a0830 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:14.795 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:14.156+0000 7fbdbea92700 1 -- 172.21.15.22:0/835006363 >> 172.21.15.22:0/835006363 conn(0x7fbdb81009c0 msgr2=0x7fbdb8102db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:14.795 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:14.156+0000 7fbdbea92700 1 -- 172.21.15.22:0/835006363 shutdown_connections 2024-03-24T09:40:14.795 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:14.156+0000 7fbdbea92700 1 -- 172.21.15.22:0/835006363 wait complete. 2024-03-24T09:40:14.796 INFO:teuthology.orchestra.run.smithi022.stdout:Deploying prometheus service with default placement... 2024-03-24T09:40:15.479 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:15 smithi022 ceph-mon[29512]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:40:15.479 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:15 smithi022 ceph-mon[29512]: Saving service ceph-exporter spec with placement * 2024-03-24T09:40:15.479 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:15 smithi022 ceph-mon[29512]: from='mgr.14120 172.21.15.22:0/3193617044' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:16.718 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-03-24T09:40:16.718 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.935+0000 7f2489af5700 1 Processor -- start 2024-03-24T09:40:16.718 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.935+0000 7f2489af5700 1 -- start start 2024-03-24T09:40:16.718 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.935+0000 7f2489af5700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24841064e0 0x7f2484106900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:16.718 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.935+0000 7f2489af5700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2484106ed0 con 0x7f24841064e0 2024-03-24T09:40:16.719 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.935+0000 7f24837fe700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24841064e0 0x7f2484106900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:16.719 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.935+0000 7f24837fe700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24841064e0 0x7f2484106900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60512/0 (socket says 172.21.15.22:60512) 2024-03-24T09:40:16.719 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.935+0000 7f24837fe700 1 -- 172.21.15.22:0/4215208862 learned_addr learned my addr 172.21.15.22:0/4215208862 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:16.719 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.935+0000 7f24837fe700 1 -- 172.21.15.22:0/4215208862 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2484107690 con 0x7f24841064e0 2024-03-24T09:40:16.719 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.936+0000 7f24837fe700 1 --2- 172.21.15.22:0/4215208862 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24841064e0 0x7f2484106900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f24740098a0 tx=0x7f2474009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88f414f8fd4bd300 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:16.719 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.936+0000 7f24827fc700 1 -- 172.21.15.22:0/4215208862 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2474004180 con 0x7f24841064e0 2024-03-24T09:40:16.719 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.936+0000 7f24827fc700 1 -- 172.21.15.22:0/4215208862 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f24740042e0 con 0x7f24841064e0 2024-03-24T09:40:16.719 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.936+0000 7f24827fc700 1 -- 172.21.15.22:0/4215208862 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2474011b10 con 0x7f24841064e0 2024-03-24T09:40:16.719 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.936+0000 7f2489af5700 1 -- 172.21.15.22:0/4215208862 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24841064e0 msgr2=0x7f2484106900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:16.719 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.936+0000 7f2489af5700 1 --2- 172.21.15.22:0/4215208862 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24841064e0 0x7f2484106900 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f24740098a0 tx=0x7f2474009bb0 comp rx=0 tx=0).stop 2024-03-24T09:40:16.719 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.936+0000 7f2489af5700 1 -- 172.21.15.22:0/4215208862 shutdown_connections 2024-03-24T09:40:16.719 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.936+0000 7f2489af5700 1 --2- 172.21.15.22:0/4215208862 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24841064e0 0x7f2484106900 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:16.720 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.936+0000 7f2489af5700 1 -- 172.21.15.22:0/4215208862 >> 172.21.15.22:0/4215208862 conn(0x7f2484101b40 msgr2=0x7f2484103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:16.720 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.937+0000 7f2489af5700 1 -- 172.21.15.22:0/4215208862 shutdown_connections 2024-03-24T09:40:16.720 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.937+0000 7f2489af5700 1 -- 172.21.15.22:0/4215208862 wait complete. 2024-03-24T09:40:16.720 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.937+0000 7f2489af5700 1 Processor -- start 2024-03-24T09:40:16.720 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.937+0000 7f2489af5700 1 -- start start 2024-03-24T09:40:16.720 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.937+0000 7f2489af5700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24841064e0 0x7f248419a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:16.720 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.937+0000 7f2489af5700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f248419a720 con 0x7f24841064e0 2024-03-24T09:40:16.720 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.938+0000 7f24837fe700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24841064e0 0x7f248419a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:16.720 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.938+0000 7f24837fe700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24841064e0 0x7f248419a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60516/0 (socket says 172.21.15.22:60516) 2024-03-24T09:40:16.720 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.938+0000 7f24837fe700 1 -- 172.21.15.22:0/2989221432 learned_addr learned my addr 172.21.15.22:0/2989221432 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:16.720 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.938+0000 7f24837fe700 1 -- 172.21.15.22:0/2989221432 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2474009550 con 0x7f24841064e0 2024-03-24T09:40:16.720 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.938+0000 7f24837fe700 1 --2- 172.21.15.22:0/2989221432 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24841064e0 0x7f248419a1e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2474009fb0 tx=0x7f2474011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:16.721 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.938+0000 7f2480ff9700 1 -- 172.21.15.22:0/2989221432 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24740038b0 con 0x7f24841064e0 2024-03-24T09:40:16.721 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.938+0000 7f2480ff9700 1 -- 172.21.15.22:0/2989221432 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2474003a10 con 0x7f24841064e0 2024-03-24T09:40:16.721 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.938+0000 7f2489af5700 1 -- 172.21.15.22:0/2989221432 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f248419a920 con 0x7f24841064e0 2024-03-24T09:40:16.721 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.938+0000 7f2480ff9700 1 -- 172.21.15.22:0/2989221432 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2474019770 con 0x7f24841064e0 2024-03-24T09:40:16.721 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.938+0000 7f2489af5700 1 -- 172.21.15.22:0/2989221432 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f248419adc0 con 0x7f24841064e0 2024-03-24T09:40:16.721 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.939+0000 7f2480ff9700 1 -- 172.21.15.22:0/2989221432 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7f24740198d0 con 0x7f24841064e0 2024-03-24T09:40:16.721 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.939+0000 7f2480ff9700 1 --2- 172.21.15.22:0/2989221432 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f246c0386e0 0x7f246c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:16.721 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.940+0000 7f2480ff9700 1 -- 172.21.15.22:0/2989221432 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f247404c7c0 con 0x7f24841064e0 2024-03-24T09:40:16.721 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.940+0000 7f2489af5700 1 -- 172.21.15.22:0/2989221432 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24840623e0 con 0x7f24841064e0 2024-03-24T09:40:16.721 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.940+0000 7f2482ffd700 1 --2- 172.21.15.22:0/2989221432 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f246c0386e0 0x7f246c03aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:16.721 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.940+0000 7f2482ffd700 1 --2- 172.21.15.22:0/2989221432 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f246c0386e0 0x7f246c03aba0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2478006fd0 tx=0x7f2478006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:16.721 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:15.945+0000 7f2480ff9700 1 -- 172.21.15.22:0/2989221432 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2474019bd0 con 0x7f24841064e0 2024-03-24T09:40:16.722 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:16.163+0000 7f2489af5700 1 -- 172.21.15.22:0/2989221432 --> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f2484102d60 con 0x7f246c0386e0 2024-03-24T09:40:16.722 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:16.168+0000 7f2480ff9700 1 -- 172.21.15.22:0/2989221432 <== mgr.14120 v2:172.21.15.22:6800/3036078499 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f2484102d60 con 0x7f246c0386e0 2024-03-24T09:40:16.722 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:16.170+0000 7f2489af5700 1 -- 172.21.15.22:0/2989221432 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f246c0386e0 msgr2=0x7f246c03aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:16.722 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:16.170+0000 7f2489af5700 1 --2- 172.21.15.22:0/2989221432 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f246c0386e0 0x7f246c03aba0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2478006fd0 tx=0x7f2478006e40 comp rx=0 tx=0).stop 2024-03-24T09:40:16.722 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:16.170+0000 7f2489af5700 1 -- 172.21.15.22:0/2989221432 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24841064e0 msgr2=0x7f248419a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:16.722 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:16.170+0000 7f2489af5700 1 --2- 172.21.15.22:0/2989221432 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24841064e0 0x7f248419a1e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2474009fb0 tx=0x7f2474011890 comp rx=0 tx=0).stop 2024-03-24T09:40:16.722 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:16.170+0000 7f2489af5700 1 -- 172.21.15.22:0/2989221432 shutdown_connections 2024-03-24T09:40:16.722 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:16.170+0000 7f2489af5700 1 --2- 172.21.15.22:0/2989221432 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f246c0386e0 0x7f246c03aba0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:16.722 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:16.170+0000 7f2489af5700 1 --2- 172.21.15.22:0/2989221432 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24841064e0 0x7f248419a1e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:16.722 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:16.170+0000 7f2489af5700 1 -- 172.21.15.22:0/2989221432 >> 172.21.15.22:0/2989221432 conn(0x7f2484101b40 msgr2=0x7f2484102650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:16.723 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:16.170+0000 7f2489af5700 1 -- 172.21.15.22:0/2989221432 shutdown_connections 2024-03-24T09:40:16.723 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:16.170+0000 7f2489af5700 1 -- 172.21.15.22:0/2989221432 wait complete. 2024-03-24T09:40:16.723 INFO:teuthology.orchestra.run.smithi022.stdout:Deploying grafana service with default placement... 2024-03-24T09:40:17.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:17 smithi022 ceph-mon[29512]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:40:17.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:17 smithi022 ceph-mon[29512]: Saving service prometheus spec with placement count:1 2024-03-24T09:40:17.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:17 smithi022 ceph-mon[29512]: from='mgr.14120 172.21.15.22:0/3193617044' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:17.907 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-03-24T09:40:17.907 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.120+0000 7feb55879700 1 Processor -- start 2024-03-24T09:40:17.907 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.120+0000 7feb55879700 1 -- start start 2024-03-24T09:40:17.907 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.120+0000 7feb55879700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feb50104dd0 0x7feb501051f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:17.907 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.120+0000 7feb55879700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb50079e10 con 0x7feb50104dd0 2024-03-24T09:40:17.907 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.121+0000 7feb4effd700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feb50104dd0 0x7feb501051f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:17.907 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.121+0000 7feb4effd700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feb50104dd0 0x7feb501051f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60530/0 (socket says 172.21.15.22:60530) 2024-03-24T09:40:17.907 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.121+0000 7feb4effd700 1 -- 172.21.15.22:0/168197542 learned_addr learned my addr 172.21.15.22:0/168197542 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:17.907 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.121+0000 7feb4effd700 1 -- 172.21.15.22:0/168197542 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb50105730 con 0x7feb50104dd0 2024-03-24T09:40:17.907 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.121+0000 7feb4effd700 1 --2- 172.21.15.22:0/168197542 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feb50104dd0 0x7feb501051f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7feb40009e80 tx=0x7feb4000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccd4f970b1380e9d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:17.908 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.121+0000 7feb4dffb700 1 -- 172.21.15.22:0/168197542 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb4000e070 con 0x7feb50104dd0 2024-03-24T09:40:17.908 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.121+0000 7feb4dffb700 1 -- 172.21.15.22:0/168197542 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feb4000b8c0 con 0x7feb50104dd0 2024-03-24T09:40:17.908 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.121+0000 7feb4dffb700 1 -- 172.21.15.22:0/168197542 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb400044d0 con 0x7feb50104dd0 2024-03-24T09:40:17.908 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.122+0000 7feb55879700 1 -- 172.21.15.22:0/168197542 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feb50104dd0 msgr2=0x7feb501051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:17.908 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.122+0000 7feb55879700 1 --2- 172.21.15.22:0/168197542 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feb50104dd0 0x7feb501051f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7feb40009e80 tx=0x7feb4000b330 comp rx=0 tx=0).stop 2024-03-24T09:40:17.908 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.122+0000 7feb55879700 1 -- 172.21.15.22:0/168197542 shutdown_connections 2024-03-24T09:40:17.908 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.122+0000 7feb55879700 1 --2- 172.21.15.22:0/168197542 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feb50104dd0 0x7feb501051f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:17.908 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.122+0000 7feb55879700 1 -- 172.21.15.22:0/168197542 >> 172.21.15.22:0/168197542 conn(0x7feb501009b0 msgr2=0x7feb50102dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:17.908 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.122+0000 7feb55879700 1 -- 172.21.15.22:0/168197542 shutdown_connections 2024-03-24T09:40:17.908 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.122+0000 7feb55879700 1 -- 172.21.15.22:0/168197542 wait complete. 2024-03-24T09:40:17.908 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.123+0000 7feb55879700 1 Processor -- start 2024-03-24T09:40:17.908 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.123+0000 7feb55879700 1 -- start start 2024-03-24T09:40:17.908 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.123+0000 7feb55879700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feb50104dd0 0x7feb5019c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:17.909 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.123+0000 7feb55879700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb5019c970 con 0x7feb50104dd0 2024-03-24T09:40:17.909 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.123+0000 7feb4effd700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feb50104dd0 0x7feb5019c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:17.909 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.123+0000 7feb4effd700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feb50104dd0 0x7feb5019c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60534/0 (socket says 172.21.15.22:60534) 2024-03-24T09:40:17.909 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.123+0000 7feb4effd700 1 -- 172.21.15.22:0/867823890 learned_addr learned my addr 172.21.15.22:0/867823890 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:17.909 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.123+0000 7feb4effd700 1 -- 172.21.15.22:0/867823890 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb40009b30 con 0x7feb50104dd0 2024-03-24T09:40:17.909 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.123+0000 7feb4effd700 1 --2- 172.21.15.22:0/867823890 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feb50104dd0 0x7feb5019c430 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7feb400072a0 tx=0x7feb4000bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:17.909 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.124+0000 7feb54877700 1 -- 172.21.15.22:0/867823890 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb4000e040 con 0x7feb50104dd0 2024-03-24T09:40:17.909 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.124+0000 7feb55879700 1 -- 172.21.15.22:0/867823890 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb5019cb70 con 0x7feb50104dd0 2024-03-24T09:40:17.909 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.124+0000 7feb54877700 1 -- 172.21.15.22:0/867823890 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feb40003dc0 con 0x7feb50104dd0 2024-03-24T09:40:17.909 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.124+0000 7feb55879700 1 -- 172.21.15.22:0/867823890 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb5019d070 con 0x7feb50104dd0 2024-03-24T09:40:17.909 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.124+0000 7feb54877700 1 -- 172.21.15.22:0/867823890 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb40012620 con 0x7feb50104dd0 2024-03-24T09:40:17.909 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.124+0000 7feb54877700 1 -- 172.21.15.22:0/867823890 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7feb40019040 con 0x7feb50104dd0 2024-03-24T09:40:17.909 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.125+0000 7feb54877700 1 --2- 172.21.15.22:0/867823890 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7feb3c0383a0 0x7feb3c03a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:17.909 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.125+0000 7feb54877700 1 -- 172.21.15.22:0/867823890 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7feb4004c530 con 0x7feb50104dd0 2024-03-24T09:40:17.910 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.125+0000 7feb4e7fc700 1 --2- 172.21.15.22:0/867823890 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7feb3c0383a0 0x7feb3c03a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:17.910 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.125+0000 7feb55879700 1 -- 172.21.15.22:0/867823890 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb50196100 con 0x7feb50104dd0 2024-03-24T09:40:17.910 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.125+0000 7feb4e7fc700 1 --2- 172.21.15.22:0/867823890 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7feb3c0383a0 0x7feb3c03a860 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7feb38006fd0 tx=0x7feb38006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:17.910 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.130+0000 7feb54877700 1 -- 172.21.15.22:0/867823890 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7feb4001f080 con 0x7feb50104dd0 2024-03-24T09:40:17.910 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.342+0000 7feb55879700 1 -- 172.21.15.22:0/867823890 --> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7feb500611f0 con 0x7feb3c0383a0 2024-03-24T09:40:17.910 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.346+0000 7feb54877700 1 -- 172.21.15.22:0/867823890 <== mgr.14120 v2:172.21.15.22:6800/3036078499 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7feb500611f0 con 0x7feb3c0383a0 2024-03-24T09:40:17.910 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.348+0000 7feb55879700 1 -- 172.21.15.22:0/867823890 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7feb3c0383a0 msgr2=0x7feb3c03a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:17.910 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.348+0000 7feb55879700 1 --2- 172.21.15.22:0/867823890 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7feb3c0383a0 0x7feb3c03a860 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7feb38006fd0 tx=0x7feb38006e40 comp rx=0 tx=0).stop 2024-03-24T09:40:17.910 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.348+0000 7feb55879700 1 -- 172.21.15.22:0/867823890 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feb50104dd0 msgr2=0x7feb5019c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:17.910 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.348+0000 7feb55879700 1 --2- 172.21.15.22:0/867823890 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feb50104dd0 0x7feb5019c430 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7feb400072a0 tx=0x7feb4000bf70 comp rx=0 tx=0).stop 2024-03-24T09:40:17.910 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.348+0000 7feb55879700 1 -- 172.21.15.22:0/867823890 shutdown_connections 2024-03-24T09:40:17.910 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.348+0000 7feb55879700 1 --2- 172.21.15.22:0/867823890 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7feb3c0383a0 0x7feb3c03a860 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:17.910 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.348+0000 7feb55879700 1 --2- 172.21.15.22:0/867823890 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feb50104dd0 0x7feb5019c430 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:17.911 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.348+0000 7feb55879700 1 -- 172.21.15.22:0/867823890 >> 172.21.15.22:0/867823890 conn(0x7feb501009b0 msgr2=0x7feb50193240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:17.911 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.349+0000 7feb55879700 1 -- 172.21.15.22:0/867823890 shutdown_connections 2024-03-24T09:40:17.911 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:17.349+0000 7feb55879700 1 -- 172.21.15.22:0/867823890 wait complete. 2024-03-24T09:40:17.911 INFO:teuthology.orchestra.run.smithi022.stdout:Deploying node-exporter service with default placement... 2024-03-24T09:40:18.543 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:18 smithi022 ceph-mon[29512]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:40:18.544 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:18 smithi022 ceph-mon[29512]: Saving service grafana spec with placement count:1 2024-03-24T09:40:18.544 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:18 smithi022 ceph-mon[29512]: from='mgr.14120 172.21.15.22:0/3193617044' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:18.544 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:18 smithi022 ceph-mon[29512]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:40:19.140 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-03-24T09:40:19.140 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.314+0000 7f27af965700 1 Processor -- start 2024-03-24T09:40:19.140 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.314+0000 7f27af965700 1 -- start start 2024-03-24T09:40:19.140 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.314+0000 7f27af965700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f27a81053f0 0x7f27a8105810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:19.140 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.314+0000 7f27af965700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a8105de0 con 0x7f27a81053f0 2024-03-24T09:40:19.141 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.314+0000 7f27ad701700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f27a81053f0 0x7f27a8105810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:19.141 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.314+0000 7f27ad701700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f27a81053f0 0x7f27a8105810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60544/0 (socket says 172.21.15.22:60544) 2024-03-24T09:40:19.141 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.314+0000 7f27ad701700 1 -- 172.21.15.22:0/30279041 learned_addr learned my addr 172.21.15.22:0/30279041 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:19.141 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.314+0000 7f27ad701700 1 -- 172.21.15.22:0/30279041 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a81065f0 con 0x7f27a81053f0 2024-03-24T09:40:19.141 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.315+0000 7f27ad701700 1 --2- 172.21.15.22:0/30279041 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f27a81053f0 0x7f27a8105810 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f27a40098a0 tx=0x7f27a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=234e0493a0e574a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:19.141 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.315+0000 7f279ffff700 1 -- 172.21.15.22:0/30279041 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27a4004180 con 0x7f27a81053f0 2024-03-24T09:40:19.141 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.315+0000 7f279ffff700 1 -- 172.21.15.22:0/30279041 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f27a40042e0 con 0x7f27a81053f0 2024-03-24T09:40:19.141 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.315+0000 7f279ffff700 1 -- 172.21.15.22:0/30279041 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27a4011b10 con 0x7f27a81053f0 2024-03-24T09:40:19.141 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.315+0000 7f27af965700 1 -- 172.21.15.22:0/30279041 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f27a81053f0 msgr2=0x7f27a8105810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:19.141 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.315+0000 7f27af965700 1 --2- 172.21.15.22:0/30279041 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f27a81053f0 0x7f27a8105810 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f27a40098a0 tx=0x7f27a4009bb0 comp rx=0 tx=0).stop 2024-03-24T09:40:19.141 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.315+0000 7f27af965700 1 -- 172.21.15.22:0/30279041 shutdown_connections 2024-03-24T09:40:19.141 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.315+0000 7f27af965700 1 --2- 172.21.15.22:0/30279041 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f27a81053f0 0x7f27a8105810 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:19.141 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.315+0000 7f27af965700 1 -- 172.21.15.22:0/30279041 >> 172.21.15.22:0/30279041 conn(0x7f27a81009b0 msgr2=0x7f27a8102dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:19.142 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.316+0000 7f27af965700 1 -- 172.21.15.22:0/30279041 shutdown_connections 2024-03-24T09:40:19.142 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.316+0000 7f27af965700 1 -- 172.21.15.22:0/30279041 wait complete. 2024-03-24T09:40:19.142 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.316+0000 7f27af965700 1 Processor -- start 2024-03-24T09:40:19.142 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.316+0000 7f27af965700 1 -- start start 2024-03-24T09:40:19.142 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.316+0000 7f27af965700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f27a81053f0 0x7f27a80799b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:19.142 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.316+0000 7f27af965700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a8079ef0 con 0x7f27a81053f0 2024-03-24T09:40:19.142 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.316+0000 7f27ad701700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f27a81053f0 0x7f27a80799b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:19.142 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.316+0000 7f27ad701700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f27a81053f0 0x7f27a80799b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60550/0 (socket says 172.21.15.22:60550) 2024-03-24T09:40:19.142 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.316+0000 7f27ad701700 1 -- 172.21.15.22:0/41555907 learned_addr learned my addr 172.21.15.22:0/41555907 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:19.142 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.317+0000 7f27ad701700 1 -- 172.21.15.22:0/41555907 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a4009550 con 0x7f27a81053f0 2024-03-24T09:40:19.142 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.317+0000 7f27ad701700 1 --2- 172.21.15.22:0/41555907 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f27a81053f0 0x7f27a80799b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f27a4009fb0 tx=0x7f27a4011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:19.142 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.317+0000 7f279e7fc700 1 -- 172.21.15.22:0/41555907 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27a40038b0 con 0x7f27a81053f0 2024-03-24T09:40:19.143 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.317+0000 7f279e7fc700 1 -- 172.21.15.22:0/41555907 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f27a4003a10 con 0x7f27a81053f0 2024-03-24T09:40:19.143 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.317+0000 7f27af965700 1 -- 172.21.15.22:0/41555907 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27a819cfd0 con 0x7f27a81053f0 2024-03-24T09:40:19.143 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.317+0000 7f279e7fc700 1 -- 172.21.15.22:0/41555907 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27a4019770 con 0x7f27a81053f0 2024-03-24T09:40:19.143 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.317+0000 7f27af965700 1 -- 172.21.15.22:0/41555907 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27a819d410 con 0x7f27a81053f0 2024-03-24T09:40:19.143 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.318+0000 7f279e7fc700 1 -- 172.21.15.22:0/41555907 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7f27a401a430 con 0x7f27a81053f0 2024-03-24T09:40:19.143 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.318+0000 7f27af965700 1 -- 172.21.15.22:0/41555907 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27a8109820 con 0x7f27a81053f0 2024-03-24T09:40:19.143 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.318+0000 7f279e7fc700 1 --2- 172.21.15.22:0/41555907 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f2794038730 0x7f279403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:19.143 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.318+0000 7f279e7fc700 1 -- 172.21.15.22:0/41555907 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f27a404c8b0 con 0x7f27a81053f0 2024-03-24T09:40:19.143 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.318+0000 7f27acf00700 1 --2- 172.21.15.22:0/41555907 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f2794038730 0x7f279403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:19.143 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.319+0000 7f27acf00700 1 --2- 172.21.15.22:0/41555907 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f2794038730 0x7f279403abf0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2798006fd0 tx=0x7f2798006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:19.143 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.323+0000 7f279e7fc700 1 -- 172.21.15.22:0/41555907 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f27a4019df0 con 0x7f27a81053f0 2024-03-24T09:40:19.143 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.539+0000 7f27af965700 1 -- 172.21.15.22:0/41555907 --> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f27a8101c70 con 0x7f2794038730 2024-03-24T09:40:19.143 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.543+0000 7f279e7fc700 1 -- 172.21.15.22:0/41555907 <== mgr.14120 v2:172.21.15.22:6800/3036078499 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f27a8101c70 con 0x7f2794038730 2024-03-24T09:40:19.143 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.545+0000 7f27af965700 1 -- 172.21.15.22:0/41555907 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f2794038730 msgr2=0x7f279403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:19.144 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.545+0000 7f27af965700 1 --2- 172.21.15.22:0/41555907 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f2794038730 0x7f279403abf0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2798006fd0 tx=0x7f2798006e40 comp rx=0 tx=0).stop 2024-03-24T09:40:19.144 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.545+0000 7f27af965700 1 -- 172.21.15.22:0/41555907 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f27a81053f0 msgr2=0x7f27a80799b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:19.144 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.545+0000 7f27af965700 1 --2- 172.21.15.22:0/41555907 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f27a81053f0 0x7f27a80799b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f27a4009fb0 tx=0x7f27a4011890 comp rx=0 tx=0).stop 2024-03-24T09:40:19.144 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.545+0000 7f27af965700 1 -- 172.21.15.22:0/41555907 shutdown_connections 2024-03-24T09:40:19.144 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.545+0000 7f27af965700 1 --2- 172.21.15.22:0/41555907 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f2794038730 0x7f279403abf0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:19.144 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.545+0000 7f27af965700 1 --2- 172.21.15.22:0/41555907 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f27a81053f0 0x7f27a80799b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:19.144 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.545+0000 7f27af965700 1 -- 172.21.15.22:0/41555907 >> 172.21.15.22:0/41555907 conn(0x7f27a81009b0 msgr2=0x7f27a8101560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:19.144 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.545+0000 7f27af965700 1 -- 172.21.15.22:0/41555907 shutdown_connections 2024-03-24T09:40:19.144 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:18.545+0000 7f27af965700 1 -- 172.21.15.22:0/41555907 wait complete. 2024-03-24T09:40:19.144 INFO:teuthology.orchestra.run.smithi022.stdout:Deploying alertmanager service with default placement... 2024-03-24T09:40:19.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:19 smithi022 ceph-mon[29512]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:40:19.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:19 smithi022 ceph-mon[29512]: Saving service node-exporter spec with placement * 2024-03-24T09:40:19.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:19 smithi022 ceph-mon[29512]: from='mgr.14120 172.21.15.22:0/3193617044' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:20.327 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-03-24T09:40:20.327 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.565+0000 7f53ee80c700 1 Processor -- start 2024-03-24T09:40:20.327 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.566+0000 7f53ee80c700 1 -- start start 2024-03-24T09:40:20.327 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.566+0000 7f53ee80c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f53e8106fb0 0x7f53e81093a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:20.327 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.566+0000 7f53ee80c700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e8079e10 con 0x7f53e8106fb0 2024-03-24T09:40:20.328 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.566+0000 7f53e7fff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f53e8106fb0 0x7f53e81093a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:20.328 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.566+0000 7f53e7fff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f53e8106fb0 0x7f53e81093a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:56064/0 (socket says 172.21.15.22:56064) 2024-03-24T09:40:20.328 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.566+0000 7f53e7fff700 1 -- 172.21.15.22:0/3741364661 learned_addr learned my addr 172.21.15.22:0/3741364661 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:20.328 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.566+0000 7f53e7fff700 1 -- 172.21.15.22:0/3741364661 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e81098e0 con 0x7f53e8106fb0 2024-03-24T09:40:20.328 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.566+0000 7f53e7fff700 1 --2- 172.21.15.22:0/3741364661 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f53e8106fb0 0x7f53e81093a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f53d00098a0 tx=0x7f53d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=685e8169078bd6a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:20.328 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.567+0000 7f53e6ffd700 1 -- 172.21.15.22:0/3741364661 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53d0004180 con 0x7f53e8106fb0 2024-03-24T09:40:20.328 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.567+0000 7f53e6ffd700 1 -- 172.21.15.22:0/3741364661 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f53d00042e0 con 0x7f53e8106fb0 2024-03-24T09:40:20.328 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.567+0000 7f53e6ffd700 1 -- 172.21.15.22:0/3741364661 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53d0011b10 con 0x7f53e8106fb0 2024-03-24T09:40:20.328 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.567+0000 7f53ee80c700 1 -- 172.21.15.22:0/3741364661 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f53e8106fb0 msgr2=0x7f53e81093a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:20.328 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.567+0000 7f53ee80c700 1 --2- 172.21.15.22:0/3741364661 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f53e8106fb0 0x7f53e81093a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f53d00098a0 tx=0x7f53d0009bb0 comp rx=0 tx=0).stop 2024-03-24T09:40:20.328 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.567+0000 7f53ee80c700 1 -- 172.21.15.22:0/3741364661 shutdown_connections 2024-03-24T09:40:20.328 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.567+0000 7f53ee80c700 1 --2- 172.21.15.22:0/3741364661 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f53e8106fb0 0x7f53e81093a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:20.328 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.567+0000 7f53ee80c700 1 -- 172.21.15.22:0/3741364661 >> 172.21.15.22:0/3741364661 conn(0x7f53e8100980 msgr2=0x7f53e8102da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:20.329 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.567+0000 7f53ee80c700 1 -- 172.21.15.22:0/3741364661 shutdown_connections 2024-03-24T09:40:20.329 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.567+0000 7f53ee80c700 1 -- 172.21.15.22:0/3741364661 wait complete. 2024-03-24T09:40:20.329 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.568+0000 7f53ee80c700 1 Processor -- start 2024-03-24T09:40:20.329 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.568+0000 7f53ee80c700 1 -- start start 2024-03-24T09:40:20.329 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.568+0000 7f53ee80c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f53e8106fb0 0x7f53e819a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:20.329 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.568+0000 7f53ee80c700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e819a750 con 0x7f53e8106fb0 2024-03-24T09:40:20.329 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.568+0000 7f53e7fff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f53e8106fb0 0x7f53e819a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:20.329 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.569+0000 7f53e7fff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f53e8106fb0 0x7f53e819a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:56074/0 (socket says 172.21.15.22:56074) 2024-03-24T09:40:20.329 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.569+0000 7f53e7fff700 1 -- 172.21.15.22:0/2365657213 learned_addr learned my addr 172.21.15.22:0/2365657213 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:20.329 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.569+0000 7f53e7fff700 1 -- 172.21.15.22:0/2365657213 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d0009550 con 0x7f53e8106fb0 2024-03-24T09:40:20.329 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.569+0000 7f53e7fff700 1 --2- 172.21.15.22:0/2365657213 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f53e8106fb0 0x7f53e819a210 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f53d0009fb0 tx=0x7f53d0011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:20.329 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.569+0000 7f53e57fa700 1 -- 172.21.15.22:0/2365657213 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53d00038b0 con 0x7f53e8106fb0 2024-03-24T09:40:20.329 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.569+0000 7f53e57fa700 1 -- 172.21.15.22:0/2365657213 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f53d0003a10 con 0x7f53e8106fb0 2024-03-24T09:40:20.330 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.569+0000 7f53ee80c700 1 -- 172.21.15.22:0/2365657213 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53e819a950 con 0x7f53e8106fb0 2024-03-24T09:40:20.330 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.569+0000 7f53e57fa700 1 -- 172.21.15.22:0/2365657213 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53d0019750 con 0x7f53e8106fb0 2024-03-24T09:40:20.330 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.569+0000 7f53ee80c700 1 -- 172.21.15.22:0/2365657213 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53e819adf0 con 0x7f53e8106fb0 2024-03-24T09:40:20.330 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.570+0000 7f53e57fa700 1 -- 172.21.15.22:0/2365657213 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7f53d00199c0 con 0x7f53e8106fb0 2024-03-24T09:40:20.330 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.570+0000 7f53e57fa700 1 --2- 172.21.15.22:0/2365657213 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f53d40386e0 0x7f53d403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:20.330 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.570+0000 7f53ee80c700 1 -- 172.21.15.22:0/2365657213 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53e810a620 con 0x7f53e8106fb0 2024-03-24T09:40:20.330 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.570+0000 7f53e57fa700 1 -- 172.21.15.22:0/2365657213 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f53d004cad0 con 0x7f53e8106fb0 2024-03-24T09:40:20.330 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.570+0000 7f53e77fe700 1 --2- 172.21.15.22:0/2365657213 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f53d40386e0 0x7f53d403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:20.330 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.571+0000 7f53e77fe700 1 --2- 172.21.15.22:0/2365657213 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f53d40386e0 0x7f53d403aba0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f53d8006fd0 tx=0x7f53d8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:20.330 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.575+0000 7f53e57fa700 1 -- 172.21.15.22:0/2365657213 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f53d0010380 con 0x7f53e8106fb0 2024-03-24T09:40:20.330 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.789+0000 7f53ee80c700 1 -- 172.21.15.22:0/2365657213 --> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f53e8101c00 con 0x7f53d40386e0 2024-03-24T09:40:20.330 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.793+0000 7f53e57fa700 1 -- 172.21.15.22:0/2365657213 <== mgr.14120 v2:172.21.15.22:6800/3036078499 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f53e8101c00 con 0x7f53d40386e0 2024-03-24T09:40:20.331 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.795+0000 7f53ee80c700 1 -- 172.21.15.22:0/2365657213 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f53d40386e0 msgr2=0x7f53d403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:20.331 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.795+0000 7f53ee80c700 1 --2- 172.21.15.22:0/2365657213 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f53d40386e0 0x7f53d403aba0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f53d8006fd0 tx=0x7f53d8006e40 comp rx=0 tx=0).stop 2024-03-24T09:40:20.331 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.795+0000 7f53ee80c700 1 -- 172.21.15.22:0/2365657213 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f53e8106fb0 msgr2=0x7f53e819a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:20.331 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.795+0000 7f53ee80c700 1 --2- 172.21.15.22:0/2365657213 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f53e8106fb0 0x7f53e819a210 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f53d0009fb0 tx=0x7f53d0011890 comp rx=0 tx=0).stop 2024-03-24T09:40:20.331 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.796+0000 7f53ee80c700 1 -- 172.21.15.22:0/2365657213 shutdown_connections 2024-03-24T09:40:20.331 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.796+0000 7f53ee80c700 1 --2- 172.21.15.22:0/2365657213 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f53d40386e0 0x7f53d403aba0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:20.331 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.796+0000 7f53ee80c700 1 --2- 172.21.15.22:0/2365657213 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f53e8106fb0 0x7f53e819a210 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:20.331 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.796+0000 7f53ee80c700 1 -- 172.21.15.22:0/2365657213 >> 172.21.15.22:0/2365657213 conn(0x7f53e8100980 msgr2=0x7f53e81014f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:20.331 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.796+0000 7f53ee80c700 1 -- 172.21.15.22:0/2365657213 shutdown_connections 2024-03-24T09:40:20.331 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:19.796+0000 7f53ee80c700 1 -- 172.21.15.22:0/2365657213 wait complete. 2024-03-24T09:40:20.947 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:20 smithi022 ceph-mon[29512]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:40:20.947 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:20 smithi022 ceph-mon[29512]: Saving service alertmanager spec with placement count:1 2024-03-24T09:40:20.947 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:20 smithi022 ceph-mon[29512]: from='mgr.14120 172.21.15.22:0/3193617044' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:20.947 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:20 smithi022 ceph-mon[29512]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:40:21.549 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.729+0000 7f80047d9700 1 Processor -- start 2024-03-24T09:40:21.549 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.729+0000 7f80047d9700 1 -- start start 2024-03-24T09:40:21.549 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.729+0000 7f80047d9700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7ffc1053e0 0x7f7ffc105800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:21.550 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.729+0000 7f80047d9700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ffc105dd0 con 0x7f7ffc1053e0 2024-03-24T09:40:21.550 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.730+0000 7f8002575700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7ffc1053e0 0x7f7ffc105800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:21.550 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.730+0000 7f8002575700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7ffc1053e0 0x7f7ffc105800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:56078/0 (socket says 172.21.15.22:56078) 2024-03-24T09:40:21.550 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.730+0000 7f8002575700 1 -- 172.21.15.22:0/88336269 learned_addr learned my addr 172.21.15.22:0/88336269 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:21.550 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.730+0000 7f8002575700 1 -- 172.21.15.22:0/88336269 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ffc1065e0 con 0x7f7ffc1053e0 2024-03-24T09:40:21.550 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.730+0000 7f8002575700 1 --2- 172.21.15.22:0/88336269 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7ffc1053e0 0x7f7ffc105800 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7ff4009e80 tx=0x7f7ff400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1ce974cdd278af5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:21.550 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.730+0000 7f8001573700 1 -- 172.21.15.22:0/88336269 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ff400e070 con 0x7f7ffc1053e0 2024-03-24T09:40:21.550 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.730+0000 7f8001573700 1 -- 172.21.15.22:0/88336269 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7ff400b8c0 con 0x7f7ffc1053e0 2024-03-24T09:40:21.550 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.730+0000 7f8001573700 1 -- 172.21.15.22:0/88336269 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ff40044d0 con 0x7f7ffc1053e0 2024-03-24T09:40:21.550 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.731+0000 7f80047d9700 1 -- 172.21.15.22:0/88336269 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7ffc1053e0 msgr2=0x7f7ffc105800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:21.550 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.731+0000 7f80047d9700 1 --2- 172.21.15.22:0/88336269 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7ffc1053e0 0x7f7ffc105800 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7ff4009e80 tx=0x7f7ff400b330 comp rx=0 tx=0).stop 2024-03-24T09:40:21.550 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.731+0000 7f80047d9700 1 -- 172.21.15.22:0/88336269 shutdown_connections 2024-03-24T09:40:21.550 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.731+0000 7f80047d9700 1 --2- 172.21.15.22:0/88336269 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7ffc1053e0 0x7f7ffc105800 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:21.551 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.731+0000 7f80047d9700 1 -- 172.21.15.22:0/88336269 >> 172.21.15.22:0/88336269 conn(0x7f7ffc100980 msgr2=0x7f7ffc102dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:21.551 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.731+0000 7f80047d9700 1 -- 172.21.15.22:0/88336269 shutdown_connections 2024-03-24T09:40:21.551 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.731+0000 7f80047d9700 1 -- 172.21.15.22:0/88336269 wait complete. 2024-03-24T09:40:21.551 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.731+0000 7f80047d9700 1 Processor -- start 2024-03-24T09:40:21.551 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.732+0000 7f80047d9700 1 -- start start 2024-03-24T09:40:21.551 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.732+0000 7f80047d9700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7ffc1053e0 0x7f7ffc1904a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:21.551 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.732+0000 7f80047d9700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ffc1909e0 con 0x7f7ffc1053e0 2024-03-24T09:40:21.551 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.732+0000 7f8002575700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7ffc1053e0 0x7f7ffc1904a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:21.551 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.732+0000 7f8002575700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7ffc1053e0 0x7f7ffc1904a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:56082/0 (socket says 172.21.15.22:56082) 2024-03-24T09:40:21.551 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.732+0000 7f8002575700 1 -- 172.21.15.22:0/460205707 learned_addr learned my addr 172.21.15.22:0/460205707 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:21.551 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.732+0000 7f8002575700 1 -- 172.21.15.22:0/460205707 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff4009b30 con 0x7f7ffc1053e0 2024-03-24T09:40:21.551 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.732+0000 7f8002575700 1 --2- 172.21.15.22:0/460205707 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7ffc1053e0 0x7f7ffc1904a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7ff40072a0 tx=0x7f7ff400bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:21.551 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.732+0000 7f7ffb7fe700 1 -- 172.21.15.22:0/460205707 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ff400e040 con 0x7f7ffc1053e0 2024-03-24T09:40:21.551 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.732+0000 7f7ffb7fe700 1 -- 172.21.15.22:0/460205707 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7ff4003dc0 con 0x7f7ffc1053e0 2024-03-24T09:40:21.551 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.732+0000 7f80047d9700 1 -- 172.21.15.22:0/460205707 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ffc18ec20 con 0x7f7ffc1053e0 2024-03-24T09:40:21.552 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.732+0000 7f7ffb7fe700 1 -- 172.21.15.22:0/460205707 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ff4012600 con 0x7f7ffc1053e0 2024-03-24T09:40:21.552 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.732+0000 7f80047d9700 1 -- 172.21.15.22:0/460205707 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ffc18f120 con 0x7f7ffc1053e0 2024-03-24T09:40:21.552 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.733+0000 7f7ffb7fe700 1 -- 172.21.15.22:0/460205707 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7f7ff4019040 con 0x7f7ffc1053e0 2024-03-24T09:40:21.552 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.733+0000 7f7ffb7fe700 1 --2- 172.21.15.22:0/460205707 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f7fe40386c0 0x7f7fe403ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:21.552 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.733+0000 7f7ffb7fe700 1 -- 172.21.15.22:0/460205707 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f7ff404c2e0 con 0x7f7ffc1053e0 2024-03-24T09:40:21.552 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.733+0000 7f80047d9700 1 -- 172.21.15.22:0/460205707 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ffc18fad0 con 0x7f7ffc1053e0 2024-03-24T09:40:21.552 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.734+0000 7f8001d74700 1 --2- 172.21.15.22:0/460205707 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f7fe40386c0 0x7f7fe403ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:21.552 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.734+0000 7f8001d74700 1 --2- 172.21.15.22:0/460205707 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f7fe40386c0 0x7f7fe403ab80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7ff0006fd0 tx=0x7f7ff0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:21.552 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.739+0000 7f7ffb7fe700 1 -- 172.21.15.22:0/460205707 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7ff401f080 con 0x7f7ffc1053e0 2024-03-24T09:40:21.552 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.942+0000 7f80047d9700 1 -- 172.21.15.22:0/460205707 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f7ffc194400 con 0x7f7ffc1053e0 2024-03-24T09:40:21.552 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.945+0000 7f7ffb7fe700 1 -- 172.21.15.22:0/460205707 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f7ff404a0d0 con 0x7f7ffc1053e0 2024-03-24T09:40:21.552 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.947+0000 7f80047d9700 1 -- 172.21.15.22:0/460205707 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f7fe40386c0 msgr2=0x7f7fe403ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:21.552 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.947+0000 7f80047d9700 1 --2- 172.21.15.22:0/460205707 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f7fe40386c0 0x7f7fe403ab80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7ff0006fd0 tx=0x7f7ff0006e40 comp rx=0 tx=0).stop 2024-03-24T09:40:21.552 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.947+0000 7f80047d9700 1 -- 172.21.15.22:0/460205707 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7ffc1053e0 msgr2=0x7f7ffc1904a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:21.552 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.947+0000 7f80047d9700 1 --2- 172.21.15.22:0/460205707 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7ffc1053e0 0x7f7ffc1904a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7ff40072a0 tx=0x7f7ff400bf70 comp rx=0 tx=0).stop 2024-03-24T09:40:21.553 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.948+0000 7f80047d9700 1 -- 172.21.15.22:0/460205707 shutdown_connections 2024-03-24T09:40:21.553 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.948+0000 7f80047d9700 1 --2- 172.21.15.22:0/460205707 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f7fe40386c0 0x7f7fe403ab80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:21.553 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.948+0000 7f80047d9700 1 --2- 172.21.15.22:0/460205707 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7ffc1053e0 0x7f7ffc1904a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:21.553 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.948+0000 7f80047d9700 1 -- 172.21.15.22:0/460205707 >> 172.21.15.22:0/460205707 conn(0x7f7ffc100980 msgr2=0x7f7ffc101840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:21.553 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.948+0000 7f80047d9700 1 -- 172.21.15.22:0/460205707 shutdown_connections 2024-03-24T09:40:21.553 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:20.948+0000 7f80047d9700 1 -- 172.21.15.22:0/460205707 wait complete. 2024-03-24T09:40:22.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:21 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/460205707' entity='client.admin' 2024-03-24T09:40:22.713 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.965+0000 7fb0e8fef700 1 Processor -- start 2024-03-24T09:40:22.713 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.965+0000 7fb0e8fef700 1 -- start start 2024-03-24T09:40:22.714 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.966+0000 7fb0e8fef700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb0e4106510 0x7fb0e4106930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:22.714 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.966+0000 7fb0e8fef700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e4106f00 con 0x7fb0e4106510 2024-03-24T09:40:22.714 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.966+0000 7fb0e259c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb0e4106510 0x7fb0e4106930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:22.714 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.966+0000 7fb0e259c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb0e4106510 0x7fb0e4106930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:56086/0 (socket says 172.21.15.22:56086) 2024-03-24T09:40:22.714 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.966+0000 7fb0e259c700 1 -- 172.21.15.22:0/137947716 learned_addr learned my addr 172.21.15.22:0/137947716 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:22.714 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.966+0000 7fb0e259c700 1 -- 172.21.15.22:0/137947716 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e4107670 con 0x7fb0e4106510 2024-03-24T09:40:22.714 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.966+0000 7fb0e259c700 1 --2- 172.21.15.22:0/137947716 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb0e4106510 0x7fb0e4106930 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb0d40098a0 tx=0x7fb0d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=488ade7307fa7740 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:22.714 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.966+0000 7fb0e159a700 1 -- 172.21.15.22:0/137947716 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0d4004180 con 0x7fb0e4106510 2024-03-24T09:40:22.714 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.966+0000 7fb0e159a700 1 -- 172.21.15.22:0/137947716 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb0d40042e0 con 0x7fb0e4106510 2024-03-24T09:40:22.714 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.966+0000 7fb0e159a700 1 -- 172.21.15.22:0/137947716 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0d4011b10 con 0x7fb0e4106510 2024-03-24T09:40:22.714 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.967+0000 7fb0e8fef700 1 -- 172.21.15.22:0/137947716 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb0e4106510 msgr2=0x7fb0e4106930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:22.714 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.967+0000 7fb0e8fef700 1 --2- 172.21.15.22:0/137947716 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb0e4106510 0x7fb0e4106930 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb0d40098a0 tx=0x7fb0d4009bb0 comp rx=0 tx=0).stop 2024-03-24T09:40:22.714 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.967+0000 7fb0e8fef700 1 -- 172.21.15.22:0/137947716 shutdown_connections 2024-03-24T09:40:22.715 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.967+0000 7fb0e8fef700 1 --2- 172.21.15.22:0/137947716 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb0e4106510 0x7fb0e4106930 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:22.715 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.967+0000 7fb0e8fef700 1 -- 172.21.15.22:0/137947716 >> 172.21.15.22:0/137947716 conn(0x7fb0e4101b10 msgr2=0x7fb0e4103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:22.715 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.967+0000 7fb0e8fef700 1 -- 172.21.15.22:0/137947716 shutdown_connections 2024-03-24T09:40:22.715 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.967+0000 7fb0e8fef700 1 -- 172.21.15.22:0/137947716 wait complete. 2024-03-24T09:40:22.715 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.968+0000 7fb0e8fef700 1 Processor -- start 2024-03-24T09:40:22.715 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.968+0000 7fb0e8fef700 1 -- start start 2024-03-24T09:40:22.715 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.968+0000 7fb0e8fef700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb0e4106510 0x7fb0e407bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:22.715 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.968+0000 7fb0e8fef700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e407c040 con 0x7fb0e4106510 2024-03-24T09:40:22.715 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.968+0000 7fb0e259c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb0e4106510 0x7fb0e407bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:22.715 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.968+0000 7fb0e259c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb0e4106510 0x7fb0e407bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:56092/0 (socket says 172.21.15.22:56092) 2024-03-24T09:40:22.715 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.968+0000 7fb0e259c700 1 -- 172.21.15.22:0/218822521 learned_addr learned my addr 172.21.15.22:0/218822521 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:22.715 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.969+0000 7fb0e259c700 1 -- 172.21.15.22:0/218822521 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d4009550 con 0x7fb0e4106510 2024-03-24T09:40:22.715 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.969+0000 7fb0e259c700 1 --2- 172.21.15.22:0/218822521 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb0e4106510 0x7fb0e407bb00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb0d40117b0 tx=0x7fb0d4011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:22.715 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.969+0000 7fb0d37fe700 1 -- 172.21.15.22:0/218822521 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0d40038b0 con 0x7fb0e4106510 2024-03-24T09:40:22.716 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.969+0000 7fb0d37fe700 1 -- 172.21.15.22:0/218822521 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb0d4003a10 con 0x7fb0e4106510 2024-03-24T09:40:22.716 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.969+0000 7fb0e8fef700 1 -- 172.21.15.22:0/218822521 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0e407a280 con 0x7fb0e4106510 2024-03-24T09:40:22.716 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.969+0000 7fb0d37fe700 1 -- 172.21.15.22:0/218822521 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0d40246e0 con 0x7fb0e4106510 2024-03-24T09:40:22.716 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.969+0000 7fb0e8fef700 1 -- 172.21.15.22:0/218822521 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0e407a720 con 0x7fb0e4106510 2024-03-24T09:40:22.716 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.970+0000 7fb0d37fe700 1 -- 172.21.15.22:0/218822521 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7fb0d402d3f0 con 0x7fb0e4106510 2024-03-24T09:40:22.716 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.970+0000 7fb0d37fe700 1 --2- 172.21.15.22:0/218822521 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fb0cc038720 0x7fb0cc03abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:22.716 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.970+0000 7fb0e8fef700 1 -- 172.21.15.22:0/218822521 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0e40623e0 con 0x7fb0e4106510 2024-03-24T09:40:22.716 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.970+0000 7fb0d37fe700 1 -- 172.21.15.22:0/218822521 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb0d404c860 con 0x7fb0e4106510 2024-03-24T09:40:22.716 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.970+0000 7fb0e1d9b700 1 --2- 172.21.15.22:0/218822521 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fb0cc038720 0x7fb0cc03abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:22.716 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.970+0000 7fb0e1d9b700 1 --2- 172.21.15.22:0/218822521 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fb0cc038720 0x7fb0cc03abe0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8006fd0 tx=0x7fb0d8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:22.716 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:21.975+0000 7fb0d37fe700 1 -- 172.21.15.22:0/218822521 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb0d4010970 con 0x7fb0e4106510 2024-03-24T09:40:22.716 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:22.176+0000 7fb0e8fef700 1 -- 172.21.15.22:0/218822521 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fb0e407b030 con 0x7fb0e4106510 2024-03-24T09:40:22.716 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:22.179+0000 7fb0d37fe700 1 -- 172.21.15.22:0/218822521 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fb0d4010380 con 0x7fb0e4106510 2024-03-24T09:40:22.716 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:22.181+0000 7fb0e8fef700 1 -- 172.21.15.22:0/218822521 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fb0cc038720 msgr2=0x7fb0cc03abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:22.717 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:22.181+0000 7fb0e8fef700 1 --2- 172.21.15.22:0/218822521 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fb0cc038720 0x7fb0cc03abe0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8006fd0 tx=0x7fb0d8006e40 comp rx=0 tx=0).stop 2024-03-24T09:40:22.717 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:22.181+0000 7fb0e8fef700 1 -- 172.21.15.22:0/218822521 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb0e4106510 msgr2=0x7fb0e407bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:22.717 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:22.181+0000 7fb0e8fef700 1 --2- 172.21.15.22:0/218822521 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb0e4106510 0x7fb0e407bb00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb0d40117b0 tx=0x7fb0d4011890 comp rx=0 tx=0).stop 2024-03-24T09:40:22.717 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:22.181+0000 7fb0e8fef700 1 -- 172.21.15.22:0/218822521 shutdown_connections 2024-03-24T09:40:22.717 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:22.181+0000 7fb0e8fef700 1 --2- 172.21.15.22:0/218822521 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7fb0cc038720 0x7fb0cc03abe0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:22.717 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:22.181+0000 7fb0e8fef700 1 --2- 172.21.15.22:0/218822521 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb0e4106510 0x7fb0e407bb00 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:22.717 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:22.181+0000 7fb0e8fef700 1 -- 172.21.15.22:0/218822521 >> 172.21.15.22:0/218822521 conn(0x7fb0e4101b10 msgr2=0x7fb0e4102670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:22.717 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:22.181+0000 7fb0e8fef700 1 -- 172.21.15.22:0/218822521 shutdown_connections 2024-03-24T09:40:22.717 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:22.181+0000 7fb0e8fef700 1 -- 172.21.15.22:0/218822521 wait complete. 2024-03-24T09:40:22.717 INFO:teuthology.orchestra.run.smithi022.stdout:Enabling the dashboard module... 2024-03-24T09:40:23.116 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:22 smithi022 ceph-mon[29512]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:40:23.116 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:22 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/218822521' entity='client.admin' 2024-03-24T09:40:24.211 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:23 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/3914440285' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-24T09:40:24.536 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.121+0000 7ff79319d700 1 Processor -- start 2024-03-24T09:40:24.536 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.121+0000 7ff79319d700 1 -- start start 2024-03-24T09:40:24.537 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.121+0000 7ff79319d700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff78c104bf0 0x7ff78c105010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:24.537 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.121+0000 7ff79319d700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff78c079e10 con 0x7ff78c104bf0 2024-03-24T09:40:24.537 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.121+0000 7ff79219b700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff78c104bf0 0x7ff78c105010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:24.537 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.122+0000 7ff79219b700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff78c104bf0 0x7ff78c105010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:56104/0 (socket says 172.21.15.22:56104) 2024-03-24T09:40:24.537 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.122+0000 7ff79219b700 1 -- 172.21.15.22:0/3867859196 learned_addr learned my addr 172.21.15.22:0/3867859196 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:24.537 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.122+0000 7ff79219b700 1 -- 172.21.15.22:0/3867859196 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff78c105550 con 0x7ff78c104bf0 2024-03-24T09:40:24.537 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.122+0000 7ff79219b700 1 --2- 172.21.15.22:0/3867859196 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff78c104bf0 0x7ff78c105010 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff7840098a0 tx=0x7ff784009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc0be453b14a6e7c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:24.537 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.122+0000 7ff791199700 1 -- 172.21.15.22:0/3867859196 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff784004180 con 0x7ff78c104bf0 2024-03-24T09:40:24.537 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.122+0000 7ff791199700 1 -- 172.21.15.22:0/3867859196 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff7840042e0 con 0x7ff78c104bf0 2024-03-24T09:40:24.537 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.122+0000 7ff791199700 1 -- 172.21.15.22:0/3867859196 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff784011b10 con 0x7ff78c104bf0 2024-03-24T09:40:24.537 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.123+0000 7ff79319d700 1 -- 172.21.15.22:0/3867859196 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff78c104bf0 msgr2=0x7ff78c105010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:24.537 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.123+0000 7ff79319d700 1 --2- 172.21.15.22:0/3867859196 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff78c104bf0 0x7ff78c105010 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff7840098a0 tx=0x7ff784009bb0 comp rx=0 tx=0).stop 2024-03-24T09:40:24.537 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.123+0000 7ff79319d700 1 -- 172.21.15.22:0/3867859196 shutdown_connections 2024-03-24T09:40:24.538 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.123+0000 7ff79319d700 1 --2- 172.21.15.22:0/3867859196 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff78c104bf0 0x7ff78c105010 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:24.538 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.123+0000 7ff79319d700 1 -- 172.21.15.22:0/3867859196 >> 172.21.15.22:0/3867859196 conn(0x7ff78c1007f0 msgr2=0x7ff78c102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:24.538 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.123+0000 7ff79319d700 1 -- 172.21.15.22:0/3867859196 shutdown_connections 2024-03-24T09:40:24.538 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.123+0000 7ff79319d700 1 -- 172.21.15.22:0/3867859196 wait complete. 2024-03-24T09:40:24.538 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.123+0000 7ff79319d700 1 Processor -- start 2024-03-24T09:40:24.538 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.123+0000 7ff79319d700 1 -- start start 2024-03-24T09:40:24.538 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.123+0000 7ff79319d700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff78c104bf0 0x7ff78c19c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:24.538 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.123+0000 7ff79319d700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff78c19c7a0 con 0x7ff78c104bf0 2024-03-24T09:40:24.538 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.124+0000 7ff79219b700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff78c104bf0 0x7ff78c19c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:24.538 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.124+0000 7ff79219b700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff78c104bf0 0x7ff78c19c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:56120/0 (socket says 172.21.15.22:56120) 2024-03-24T09:40:24.538 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.124+0000 7ff79219b700 1 -- 172.21.15.22:0/3914440285 learned_addr learned my addr 172.21.15.22:0/3914440285 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:24.538 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.124+0000 7ff79219b700 1 -- 172.21.15.22:0/3914440285 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff784009550 con 0x7ff78c104bf0 2024-03-24T09:40:24.538 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.124+0000 7ff79219b700 1 --2- 172.21.15.22:0/3914440285 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff78c104bf0 0x7ff78c19c260 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff7840117b0 tx=0x7ff784011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:24.539 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.124+0000 7ff78b7fe700 1 -- 172.21.15.22:0/3914440285 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7840038b0 con 0x7ff78c104bf0 2024-03-24T09:40:24.539 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.124+0000 7ff78b7fe700 1 -- 172.21.15.22:0/3914440285 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff784003a10 con 0x7ff78c104bf0 2024-03-24T09:40:24.539 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.124+0000 7ff79319d700 1 -- 172.21.15.22:0/3914440285 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff78c19c9a0 con 0x7ff78c104bf0 2024-03-24T09:40:24.539 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.124+0000 7ff78b7fe700 1 -- 172.21.15.22:0/3914440285 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff784024780 con 0x7ff78c104bf0 2024-03-24T09:40:24.539 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.124+0000 7ff79319d700 1 -- 172.21.15.22:0/3914440285 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff78c19ce40 con 0x7ff78c104bf0 2024-03-24T09:40:24.539 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.125+0000 7ff78b7fe700 1 -- 172.21.15.22:0/3914440285 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 8) v1 ==== 45548+0+0 (secure 0 0 0) 0x7ff7840248e0 con 0x7ff78c104bf0 2024-03-24T09:40:24.539 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.125+0000 7ff79319d700 1 -- 172.21.15.22:0/3914440285 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff78c195f10 con 0x7ff78c104bf0 2024-03-24T09:40:24.539 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.125+0000 7ff78b7fe700 1 --2- 172.21.15.22:0/3914440285 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7ff774038770 0x7ff77403ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:24.539 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.125+0000 7ff78b7fe700 1 -- 172.21.15.22:0/3914440285 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff78404c7f0 con 0x7ff78c104bf0 2024-03-24T09:40:24.539 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.125+0000 7ff79199a700 1 --2- 172.21.15.22:0/3914440285 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7ff774038770 0x7ff77403ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:24.539 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.126+0000 7ff79199a700 1 --2- 172.21.15.22:0/3914440285 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7ff774038770 0x7ff77403ac30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff780006fd0 tx=0x7ff780006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:24.539 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.130+0000 7ff78b7fe700 1 -- 172.21.15.22:0/3914440285 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff784024b90 con 0x7ff78c104bf0 2024-03-24T09:40:24.539 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.388+0000 7ff79319d700 1 -- 172.21.15.22:0/3914440285 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7ff78c02ccb0 con 0x7ff78c104bf0 2024-03-24T09:40:24.539 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.951+0000 7ff78b7fe700 1 -- 172.21.15.22:0/3914440285 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mgrmap(e 9) v1 ==== 45561+0+0 (secure 0 0 0) 0x7ff78401ac10 con 0x7ff78c104bf0 2024-03-24T09:40:24.540 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.953+0000 7ff78b7fe700 1 -- 172.21.15.22:0/3914440285 <== mon.0 v2:172.21.15.22:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7ff78401a960 con 0x7ff78c104bf0 2024-03-24T09:40:24.540 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.955+0000 7ff79319d700 1 -- 172.21.15.22:0/3914440285 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7ff774038770 msgr2=0x7ff77403ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:24.540 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.955+0000 7ff79319d700 1 --2- 172.21.15.22:0/3914440285 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7ff774038770 0x7ff77403ac30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff780006fd0 tx=0x7ff780006e40 comp rx=0 tx=0).stop 2024-03-24T09:40:24.540 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.955+0000 7ff79319d700 1 -- 172.21.15.22:0/3914440285 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff78c104bf0 msgr2=0x7ff78c19c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:24.540 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.955+0000 7ff79319d700 1 --2- 172.21.15.22:0/3914440285 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff78c104bf0 0x7ff78c19c260 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff7840117b0 tx=0x7ff784011890 comp rx=0 tx=0).stop 2024-03-24T09:40:24.540 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.955+0000 7ff79319d700 1 -- 172.21.15.22:0/3914440285 shutdown_connections 2024-03-24T09:40:24.540 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.956+0000 7ff79319d700 1 --2- 172.21.15.22:0/3914440285 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7ff774038770 0x7ff77403ac30 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:24.540 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.956+0000 7ff79319d700 1 --2- 172.21.15.22:0/3914440285 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff78c104bf0 0x7ff78c19c260 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:24.540 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.956+0000 7ff79319d700 1 -- 172.21.15.22:0/3914440285 >> 172.21.15.22:0/3914440285 conn(0x7ff78c1007f0 msgr2=0x7ff78c193050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:24.540 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.956+0000 7ff79319d700 1 -- 172.21.15.22:0/3914440285 shutdown_connections 2024-03-24T09:40:24.540 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:23.956+0000 7ff79319d700 1 -- 172.21.15.22:0/3914440285 wait complete. 2024-03-24T09:40:25.186 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:24 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/3914440285' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-24T09:40:25.186 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:24 smithi022 ceph-mon[29512]: mgrmap e9: smithi022.emgmxz(active, since 25s) 2024-03-24T09:40:25.835 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout { 2024-03-24T09:40:25.835 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-24T09:40:25.835 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T09:40:25.835 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "active_name": "smithi022.emgmxz", 2024-03-24T09:40:25.835 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-24T09:40:25.835 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout } 2024-03-24T09:40:25.836 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.982+0000 7efde2540700 1 Processor -- start 2024-03-24T09:40:25.836 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.982+0000 7efde2540700 1 -- start start 2024-03-24T09:40:25.836 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.982+0000 7efde2540700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efddc1053c0 0x7efddc1057e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:25.836 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.982+0000 7efde2540700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efddc105db0 con 0x7efddc1053c0 2024-03-24T09:40:25.836 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.982+0000 7efddbfff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efddc1053c0 0x7efddc1057e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:25.836 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.982+0000 7efddbfff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efddc1053c0 0x7efddc1057e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:56140/0 (socket says 172.21.15.22:56140) 2024-03-24T09:40:25.836 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.982+0000 7efddbfff700 1 -- 172.21.15.22:0/667037345 learned_addr learned my addr 172.21.15.22:0/667037345 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:25.836 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.982+0000 7efddbfff700 1 -- 172.21.15.22:0/667037345 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efddc1065c0 con 0x7efddc1053c0 2024-03-24T09:40:25.836 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.983+0000 7efddbfff700 1 --2- 172.21.15.22:0/667037345 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efddc1053c0 0x7efddc1057e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7efdcc0098a0 tx=0x7efdcc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a49603f0d763eb08 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:25.836 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.983+0000 7efddaffd700 1 -- 172.21.15.22:0/667037345 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdcc004180 con 0x7efddc1053c0 2024-03-24T09:40:25.836 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.983+0000 7efddaffd700 1 -- 172.21.15.22:0/667037345 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efdcc0042e0 con 0x7efddc1053c0 2024-03-24T09:40:25.836 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.983+0000 7efddaffd700 1 -- 172.21.15.22:0/667037345 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdcc011b10 con 0x7efddc1053c0 2024-03-24T09:40:25.836 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.983+0000 7efde2540700 1 -- 172.21.15.22:0/667037345 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efddc1053c0 msgr2=0x7efddc1057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:25.836 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.983+0000 7efde2540700 1 --2- 172.21.15.22:0/667037345 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efddc1053c0 0x7efddc1057e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7efdcc0098a0 tx=0x7efdcc009bb0 comp rx=0 tx=0).stop 2024-03-24T09:40:25.837 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.983+0000 7efde2540700 1 -- 172.21.15.22:0/667037345 shutdown_connections 2024-03-24T09:40:25.837 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.983+0000 7efde2540700 1 --2- 172.21.15.22:0/667037345 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efddc1053c0 0x7efddc1057e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:25.837 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.983+0000 7efde2540700 1 -- 172.21.15.22:0/667037345 >> 172.21.15.22:0/667037345 conn(0x7efddc1009c0 msgr2=0x7efddc102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:25.837 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.983+0000 7efde2540700 1 -- 172.21.15.22:0/667037345 shutdown_connections 2024-03-24T09:40:25.837 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.983+0000 7efde2540700 1 -- 172.21.15.22:0/667037345 wait complete. 2024-03-24T09:40:25.837 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.984+0000 7efde2540700 1 Processor -- start 2024-03-24T09:40:25.837 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.984+0000 7efde2540700 1 -- start start 2024-03-24T09:40:25.837 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.984+0000 7efde2540700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efddc1053c0 0x7efddc19a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:25.837 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.984+0000 7efde2540700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efddc19a7a0 con 0x7efddc1053c0 2024-03-24T09:40:25.837 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.984+0000 7efddbfff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efddc1053c0 0x7efddc19a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:25.837 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.985+0000 7efddbfff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efddc1053c0 0x7efddc19a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:56146/0 (socket says 172.21.15.22:56146) 2024-03-24T09:40:25.837 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.985+0000 7efddbfff700 1 -- 172.21.15.22:0/2495323067 learned_addr learned my addr 172.21.15.22:0/2495323067 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:25.837 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.985+0000 7efddbfff700 1 -- 172.21.15.22:0/2495323067 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdcc009550 con 0x7efddc1053c0 2024-03-24T09:40:25.837 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.985+0000 7efddbfff700 1 --2- 172.21.15.22:0/2495323067 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efddc1053c0 0x7efddc19a260 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7efdcc0117b0 tx=0x7efdcc011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:25.838 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.985+0000 7efdd97fa700 1 -- 172.21.15.22:0/2495323067 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdcc0038b0 con 0x7efddc1053c0 2024-03-24T09:40:25.838 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.985+0000 7efdd97fa700 1 -- 172.21.15.22:0/2495323067 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efdcc003a10 con 0x7efddc1053c0 2024-03-24T09:40:25.838 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.985+0000 7efde2540700 1 -- 172.21.15.22:0/2495323067 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efddc19a9a0 con 0x7efddc1053c0 2024-03-24T09:40:25.838 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.985+0000 7efdd97fa700 1 -- 172.21.15.22:0/2495323067 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdcc0246e0 con 0x7efddc1053c0 2024-03-24T09:40:25.838 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.985+0000 7efde2540700 1 -- 172.21.15.22:0/2495323067 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efddc19ae40 con 0x7efddc1053c0 2024-03-24T09:40:25.838 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.986+0000 7efdd97fa700 1 -- 172.21.15.22:0/2495323067 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 9) v1 ==== 45561+0+0 (secure 0 0 0) 0x7efdcc02d3f0 con 0x7efddc1053c0 2024-03-24T09:40:25.838 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.986+0000 7efdd97fa700 1 --2- 172.21.15.22:0/2495323067 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7efdc40387c0 0x7efdc403ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:25.838 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.986+0000 7efdd97fa700 1 -- 172.21.15.22:0/2495323067 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7efdcc04d830 con 0x7efddc1053c0 2024-03-24T09:40:25.838 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.986+0000 7efddb7fe700 1 -- 172.21.15.22:0/2495323067 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7efdc40387c0 msgr2=0x7efdc403ac80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/3036078499 2024-03-24T09:40:25.838 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.986+0000 7efddb7fe700 1 --2- 172.21.15.22:0/2495323067 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7efdc40387c0 0x7efdc403ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T09:40:25.838 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.986+0000 7efde2540700 1 -- 172.21.15.22:0/2495323067 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efddc194400 con 0x7efddc1053c0 2024-03-24T09:40:25.838 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:24.991+0000 7efdd97fa700 1 -- 172.21.15.22:0/2495323067 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7efdcc019460 con 0x7efddc1053c0 2024-03-24T09:40:25.839 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:25.187+0000 7efddb7fe700 1 -- 172.21.15.22:0/2495323067 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7efdc40387c0 msgr2=0x7efdc403ac80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/3036078499 2024-03-24T09:40:25.839 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:25.187+0000 7efddb7fe700 1 --2- 172.21.15.22:0/2495323067 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7efdc40387c0 0x7efdc403ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T09:40:25.839 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:25.251+0000 7efde2540700 1 -- 172.21.15.22:0/2495323067 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7efddc0619c0 con 0x7efddc1053c0 2024-03-24T09:40:25.839 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:25.251+0000 7efdd97fa700 1 -- 172.21.15.22:0/2495323067 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7efdcc010380 con 0x7efddc1053c0 2024-03-24T09:40:25.839 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:25.253+0000 7efde2540700 1 -- 172.21.15.22:0/2495323067 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7efdc40387c0 msgr2=0x7efdc403ac80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T09:40:25.839 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:25.253+0000 7efde2540700 1 --2- 172.21.15.22:0/2495323067 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7efdc40387c0 0x7efdc403ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:25.839 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:25.253+0000 7efde2540700 1 -- 172.21.15.22:0/2495323067 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efddc1053c0 msgr2=0x7efddc19a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:25.839 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:25.253+0000 7efde2540700 1 --2- 172.21.15.22:0/2495323067 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efddc1053c0 0x7efddc19a260 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7efdcc0117b0 tx=0x7efdcc011890 comp rx=0 tx=0).stop 2024-03-24T09:40:25.839 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:25.253+0000 7efde2540700 1 -- 172.21.15.22:0/2495323067 shutdown_connections 2024-03-24T09:40:25.839 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:25.253+0000 7efde2540700 1 --2- 172.21.15.22:0/2495323067 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7efdc40387c0 0x7efdc403ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:25.839 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:25.253+0000 7efde2540700 1 --2- 172.21.15.22:0/2495323067 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efddc1053c0 0x7efddc19a260 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:25.839 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:25.253+0000 7efde2540700 1 -- 172.21.15.22:0/2495323067 >> 172.21.15.22:0/2495323067 conn(0x7efddc1009c0 msgr2=0x7efddc1018b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:25.839 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:25.253+0000 7efde2540700 1 -- 172.21.15.22:0/2495323067 shutdown_connections 2024-03-24T09:40:25.839 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:25.253+0000 7efde2540700 1 -- 172.21.15.22:0/2495323067 wait complete. 2024-03-24T09:40:25.840 INFO:teuthology.orchestra.run.smithi022.stdout:Waiting for the mgr to restart... 2024-03-24T09:40:25.840 INFO:teuthology.orchestra.run.smithi022.stdout:Waiting for mgr epoch 9... 2024-03-24T09:40:26.112 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:26 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/2495323067' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-24T09:40:31.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:31 smithi022 ceph-mon[29512]: Active manager daemon smithi022.emgmxz restarted 2024-03-24T09:40:31.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:31 smithi022 ceph-mon[29512]: Activating manager daemon smithi022.emgmxz 2024-03-24T09:40:31.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:31 smithi022 ceph-mon[29512]: osdmap e3: 0 total, 0 up, 0 in 2024-03-24T09:40:31.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:31 smithi022 ceph-mon[29512]: mgrmap e10: smithi022.emgmxz(active, starting, since 0.00315212s) 2024-03-24T09:40:31.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:31 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mon metadata", "id": "smithi022"}]: dispatch 2024-03-24T09:40:31.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:31 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mgr metadata", "who": "smithi022.emgmxz", "id": "smithi022.emgmxz"}]: dispatch 2024-03-24T09:40:31.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:31 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T09:40:31.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:31 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T09:40:31.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:31 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T09:40:31.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:31 smithi022 ceph-mon[29512]: Manager daemon smithi022.emgmxz is now available 2024-03-24T09:40:31.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:31 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:40:31.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:31 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:40:31.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:31 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.emgmxz/mirror_snapshot_schedule"}]: dispatch 2024-03-24T09:40:31.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:31 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.emgmxz/trash_purge_schedule"}]: dispatch 2024-03-24T09:40:32.647 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout { 2024-03-24T09:40:32.647 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-24T09:40:32.647 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-24T09:40:32.647 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout } 2024-03-24T09:40:32.648 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.234+0000 7f39f3973700 1 Processor -- start 2024-03-24T09:40:32.648 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.234+0000 7f39f3973700 1 -- start start 2024-03-24T09:40:32.648 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.235+0000 7f39f3973700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f39ec1064e0 0x7f39ec106900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:32.648 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.235+0000 7f39f3973700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39ec106ed0 con 0x7f39ec1064e0 2024-03-24T09:40:32.648 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.235+0000 7f39f170f700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f39ec1064e0 0x7f39ec106900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:32.648 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.235+0000 7f39f170f700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f39ec1064e0 0x7f39ec106900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:56150/0 (socket says 172.21.15.22:56150) 2024-03-24T09:40:32.648 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.235+0000 7f39f170f700 1 -- 172.21.15.22:0/263317924 learned_addr learned my addr 172.21.15.22:0/263317924 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:32.648 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.235+0000 7f39f170f700 1 -- 172.21.15.22:0/263317924 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39ec107690 con 0x7f39ec1064e0 2024-03-24T09:40:32.648 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.235+0000 7f39f170f700 1 --2- 172.21.15.22:0/263317924 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f39ec1064e0 0x7f39ec106900 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f39e80098a0 tx=0x7f39e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ed389dba8b3dc0e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:32.649 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.235+0000 7f39dffff700 1 -- 172.21.15.22:0/263317924 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39e8004180 con 0x7f39ec1064e0 2024-03-24T09:40:32.649 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.235+0000 7f39dffff700 1 -- 172.21.15.22:0/263317924 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f39e80042e0 con 0x7f39ec1064e0 2024-03-24T09:40:32.649 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.236+0000 7f39dffff700 1 -- 172.21.15.22:0/263317924 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39e8011b10 con 0x7f39ec1064e0 2024-03-24T09:40:32.649 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.236+0000 7f39f3973700 1 -- 172.21.15.22:0/263317924 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f39ec1064e0 msgr2=0x7f39ec106900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:32.649 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.236+0000 7f39f3973700 1 --2- 172.21.15.22:0/263317924 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f39ec1064e0 0x7f39ec106900 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f39e80098a0 tx=0x7f39e8009bb0 comp rx=0 tx=0).stop 2024-03-24T09:40:32.649 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.236+0000 7f39f3973700 1 -- 172.21.15.22:0/263317924 shutdown_connections 2024-03-24T09:40:32.649 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.236+0000 7f39f3973700 1 --2- 172.21.15.22:0/263317924 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f39ec1064e0 0x7f39ec106900 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:32.649 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.236+0000 7f39f3973700 1 -- 172.21.15.22:0/263317924 >> 172.21.15.22:0/263317924 conn(0x7f39ec101b40 msgr2=0x7f39ec103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:32.649 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.236+0000 7f39f3973700 1 -- 172.21.15.22:0/263317924 shutdown_connections 2024-03-24T09:40:32.649 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.236+0000 7f39f3973700 1 -- 172.21.15.22:0/263317924 wait complete. 2024-03-24T09:40:32.650 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.237+0000 7f39f3973700 1 Processor -- start 2024-03-24T09:40:32.650 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.237+0000 7f39f3973700 1 -- start start 2024-03-24T09:40:32.650 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.237+0000 7f39f3973700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f39ec1064e0 0x7f39ec07bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:32.650 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.237+0000 7f39f3973700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39ec07c040 con 0x7f39ec1064e0 2024-03-24T09:40:32.650 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.237+0000 7f39f170f700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f39ec1064e0 0x7f39ec07bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:32.650 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.237+0000 7f39f170f700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f39ec1064e0 0x7f39ec07bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:56166/0 (socket says 172.21.15.22:56166) 2024-03-24T09:40:32.650 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.237+0000 7f39f170f700 1 -- 172.21.15.22:0/103384057 learned_addr learned my addr 172.21.15.22:0/103384057 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:32.650 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.238+0000 7f39f170f700 1 -- 172.21.15.22:0/103384057 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e8009550 con 0x7f39ec1064e0 2024-03-24T09:40:32.650 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.238+0000 7f39f170f700 1 --2- 172.21.15.22:0/103384057 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f39ec1064e0 0x7f39ec07bb00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f39e80117b0 tx=0x7f39e8011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:32.651 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.238+0000 7f39de7fc700 1 -- 172.21.15.22:0/103384057 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39e80038b0 con 0x7f39ec1064e0 2024-03-24T09:40:32.651 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.238+0000 7f39de7fc700 1 -- 172.21.15.22:0/103384057 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f39e8003a10 con 0x7f39ec1064e0 2024-03-24T09:40:32.651 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.238+0000 7f39f3973700 1 -- 172.21.15.22:0/103384057 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39ec07a280 con 0x7f39ec1064e0 2024-03-24T09:40:32.651 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.238+0000 7f39de7fc700 1 -- 172.21.15.22:0/103384057 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39e80246e0 con 0x7f39ec1064e0 2024-03-24T09:40:32.651 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.238+0000 7f39f3973700 1 -- 172.21.15.22:0/103384057 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39ec07a720 con 0x7f39ec1064e0 2024-03-24T09:40:32.651 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.239+0000 7f39de7fc700 1 -- 172.21.15.22:0/103384057 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 9) v1 ==== 45561+0+0 (secure 0 0 0) 0x7f39e801a430 con 0x7f39ec1064e0 2024-03-24T09:40:32.651 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.239+0000 7f39de7fc700 1 --2- 172.21.15.22:0/103384057 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f39d80387c0 0x7f39d803ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:32.651 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.239+0000 7f39de7fc700 1 -- 172.21.15.22:0/103384057 --> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f39d803b390 con 0x7f39d80387c0 2024-03-24T09:40:32.651 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.239+0000 7f39f0f0e700 1 -- 172.21.15.22:0/103384057 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f39d80387c0 msgr2=0x7f39d803ac80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/3036078499 2024-03-24T09:40:32.652 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.239+0000 7f39de7fc700 1 -- 172.21.15.22:0/103384057 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f39e804c970 con 0x7f39ec1064e0 2024-03-24T09:40:32.652 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.239+0000 7f39f0f0e700 1 --2- 172.21.15.22:0/103384057 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f39d80387c0 0x7f39d803ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T09:40:32.652 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.440+0000 7f39f0f0e700 1 -- 172.21.15.22:0/103384057 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f39d80387c0 msgr2=0x7f39d803ac80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/3036078499 2024-03-24T09:40:32.652 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.440+0000 7f39f0f0e700 1 --2- 172.21.15.22:0/103384057 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f39d80387c0 0x7f39d803ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T09:40:32.652 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.841+0000 7f39f0f0e700 1 -- 172.21.15.22:0/103384057 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f39d80387c0 msgr2=0x7f39d803ac80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/3036078499 2024-03-24T09:40:32.652 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:26.841+0000 7f39f0f0e700 1 --2- 172.21.15.22:0/103384057 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f39d80387c0 0x7f39d803ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T09:40:32.652 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:27.642+0000 7f39f0f0e700 1 -- 172.21.15.22:0/103384057 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f39d80387c0 msgr2=0x7f39d803ac80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/3036078499 2024-03-24T09:40:32.652 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:27.642+0000 7f39f0f0e700 1 --2- 172.21.15.22:0/103384057 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f39d80387c0 0x7f39d803ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T09:40:32.652 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:29.243+0000 7f39f0f0e700 1 -- 172.21.15.22:0/103384057 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f39d80387c0 msgr2=0x7f39d803ac80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/3036078499 2024-03-24T09:40:32.652 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:29.243+0000 7f39f0f0e700 1 --2- 172.21.15.22:0/103384057 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f39d80387c0 0x7f39d803ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T09:40:32.653 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:30.947+0000 7f39de7fc700 1 -- 172.21.15.22:0/103384057 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mgrmap(e 10) v1 ==== 45328+0+0 (secure 0 0 0) 0x7f39e801ab80 con 0x7f39ec1064e0 2024-03-24T09:40:32.653 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:30.947+0000 7f39de7fc700 1 -- 172.21.15.22:0/103384057 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f39d80387c0 msgr2=0x7f39d803ac80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T09:40:32.653 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:30.947+0000 7f39de7fc700 1 --2- 172.21.15.22:0/103384057 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f39d80387c0 0x7f39d803ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:32.653 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:31.948+0000 7f39de7fc700 1 -- 172.21.15.22:0/103384057 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mgrmap(e 11) v1 ==== 45455+0+0 (secure 0 0 0) 0x7f39e804e420 con 0x7f39ec1064e0 2024-03-24T09:40:32.653 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:31.948+0000 7f39de7fc700 1 --2- 172.21.15.22:0/103384057 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f39d803bfc0 0x7f39d803e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:32.653 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:31.948+0000 7f39de7fc700 1 -- 172.21.15.22:0/103384057 --> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f39d803b390 con 0x7f39d803bfc0 2024-03-24T09:40:32.653 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:31.948+0000 7f39f0f0e700 1 --2- 172.21.15.22:0/103384057 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f39d803bfc0 0x7f39d803e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:32.653 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:31.949+0000 7f39f0f0e700 1 --2- 172.21.15.22:0/103384057 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f39d803bfc0 0x7f39d803e3b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f39e0003a10 tx=0x7f39e0009270 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:32.653 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:31.949+0000 7f39de7fc700 1 -- 172.21.15.22:0/103384057 <== mgr.14164 v2:172.21.15.22:6800/2979399337 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f39d803b390 con 0x7f39d803bfc0 2024-03-24T09:40:32.653 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:31.955+0000 7f39f3973700 1 -- 172.21.15.22:0/103384057 --> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f39ec1084f0 con 0x7f39d803bfc0 2024-03-24T09:40:32.654 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:31.956+0000 7f39de7fc700 1 -- 172.21.15.22:0/103384057 <== mgr.14164 v2:172.21.15.22:6800/2979399337 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f39e000c9c0 con 0x7f39d803bfc0 2024-03-24T09:40:32.654 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:31.956+0000 7f39f3973700 1 -- 172.21.15.22:0/103384057 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f39d803bfc0 msgr2=0x7f39d803e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:32.654 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:31.956+0000 7f39f3973700 1 --2- 172.21.15.22:0/103384057 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f39d803bfc0 0x7f39d803e3b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f39e0003a10 tx=0x7f39e0009270 comp rx=0 tx=0).stop 2024-03-24T09:40:32.654 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:31.956+0000 7f39f3973700 1 -- 172.21.15.22:0/103384057 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f39ec1064e0 msgr2=0x7f39ec07bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:32.654 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:31.956+0000 7f39f3973700 1 --2- 172.21.15.22:0/103384057 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f39ec1064e0 0x7f39ec07bb00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f39e80117b0 tx=0x7f39e8011890 comp rx=0 tx=0).stop 2024-03-24T09:40:32.654 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:31.956+0000 7f39f3973700 1 -- 172.21.15.22:0/103384057 shutdown_connections 2024-03-24T09:40:32.654 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:31.956+0000 7f39f3973700 1 --2- 172.21.15.22:0/103384057 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f39d803bfc0 0x7f39d803e3b0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:32.654 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:31.956+0000 7f39f3973700 1 --2- 172.21.15.22:0/103384057 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f39ec1064e0 0x7f39ec07bb00 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:32.654 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:31.956+0000 7f39f3973700 1 --2- 172.21.15.22:0/103384057 >> [v2:172.21.15.22:6800/3036078499,v1:172.21.15.22:6801/3036078499] conn(0x7f39d80387c0 0x7f39d803ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:32.654 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:31.956+0000 7f39f3973700 1 -- 172.21.15.22:0/103384057 >> 172.21.15.22:0/103384057 conn(0x7f39ec101b40 msgr2=0x7f39ec102650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:32.655 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:31.956+0000 7f39f3973700 1 -- 172.21.15.22:0/103384057 shutdown_connections 2024-03-24T09:40:32.655 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:31.956+0000 7f39f3973700 1 -- 172.21.15.22:0/103384057 wait complete. 2024-03-24T09:40:32.655 INFO:teuthology.orchestra.run.smithi022.stdout:mgr epoch 9 is available 2024-03-24T09:40:32.655 INFO:teuthology.orchestra.run.smithi022.stdout:Generating a dashboard self-signed certificate... 2024-03-24T09:40:33.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:32 smithi022 ceph-mon[29512]: mgrmap e11: smithi022.emgmxz(active, since 1.00609s) 2024-03-24T09:40:33.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:32 smithi022 ceph-mon[29512]: [24/Mar/2024:09:40:31] ENGINE Bus STARTING 2024-03-24T09:40:33.984 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-24T09:40:33.984 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.116+0000 7f087b9fb700 1 Processor -- start 2024-03-24T09:40:33.984 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.116+0000 7f087b9fb700 1 -- start start 2024-03-24T09:40:33.984 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.116+0000 7f087b9fb700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f08741053e0 0x7f0874105800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:33.984 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.116+0000 7f087b9fb700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0874105dd0 con 0x7f08741053e0 2024-03-24T09:40:33.984 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.116+0000 7f0879797700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f08741053e0 0x7f0874105800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:33.984 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.116+0000 7f0879797700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f08741053e0 0x7f0874105800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:34706/0 (socket says 172.21.15.22:34706) 2024-03-24T09:40:33.984 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.116+0000 7f0879797700 1 -- 172.21.15.22:0/1328316468 learned_addr learned my addr 172.21.15.22:0/1328316468 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:33.985 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.116+0000 7f0879797700 1 -- 172.21.15.22:0/1328316468 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08741065e0 con 0x7f08741053e0 2024-03-24T09:40:33.985 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.117+0000 7f0879797700 1 --2- 172.21.15.22:0/1328316468 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f08741053e0 0x7f0874105800 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f08700098a0 tx=0x7f0870009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c903651cd7bc2e6a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:33.985 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.117+0000 7f086bfff700 1 -- 172.21.15.22:0/1328316468 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0870004180 con 0x7f08741053e0 2024-03-24T09:40:33.985 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.117+0000 7f086bfff700 1 -- 172.21.15.22:0/1328316468 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f08700042e0 con 0x7f08741053e0 2024-03-24T09:40:33.985 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.117+0000 7f086bfff700 1 -- 172.21.15.22:0/1328316468 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0870011b10 con 0x7f08741053e0 2024-03-24T09:40:33.985 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.118+0000 7f087b9fb700 1 -- 172.21.15.22:0/1328316468 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f08741053e0 msgr2=0x7f0874105800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:33.985 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.118+0000 7f087b9fb700 1 --2- 172.21.15.22:0/1328316468 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f08741053e0 0x7f0874105800 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f08700098a0 tx=0x7f0870009bb0 comp rx=0 tx=0).stop 2024-03-24T09:40:33.985 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.118+0000 7f087b9fb700 1 -- 172.21.15.22:0/1328316468 shutdown_connections 2024-03-24T09:40:33.985 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.118+0000 7f087b9fb700 1 --2- 172.21.15.22:0/1328316468 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f08741053e0 0x7f0874105800 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:33.985 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.118+0000 7f087b9fb700 1 -- 172.21.15.22:0/1328316468 >> 172.21.15.22:0/1328316468 conn(0x7f0874100960 msgr2=0x7f0874102dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:33.985 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.118+0000 7f087b9fb700 1 -- 172.21.15.22:0/1328316468 shutdown_connections 2024-03-24T09:40:33.986 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.118+0000 7f087b9fb700 1 -- 172.21.15.22:0/1328316468 wait complete. 2024-03-24T09:40:33.986 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.119+0000 7f087b9fb700 1 Processor -- start 2024-03-24T09:40:33.986 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.119+0000 7f087b9fb700 1 -- start start 2024-03-24T09:40:33.986 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.119+0000 7f087b9fb700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f08741053e0 0x7f087419a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:33.986 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.119+0000 7f087b9fb700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087419a820 con 0x7f08741053e0 2024-03-24T09:40:33.986 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.119+0000 7f0879797700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f08741053e0 0x7f087419a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:33.986 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.119+0000 7f0879797700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f08741053e0 0x7f087419a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:34710/0 (socket says 172.21.15.22:34710) 2024-03-24T09:40:33.986 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.119+0000 7f0879797700 1 -- 172.21.15.22:0/2504339695 learned_addr learned my addr 172.21.15.22:0/2504339695 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:33.986 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.119+0000 7f0879797700 1 -- 172.21.15.22:0/2504339695 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0870009550 con 0x7f08741053e0 2024-03-24T09:40:33.986 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.119+0000 7f0879797700 1 --2- 172.21.15.22:0/2504339695 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f08741053e0 0x7f087419a2e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f08700117b0 tx=0x7f0870011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:33.986 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.119+0000 7f086a7fc700 1 -- 172.21.15.22:0/2504339695 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f08700038b0 con 0x7f08741053e0 2024-03-24T09:40:33.986 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.119+0000 7f086a7fc700 1 -- 172.21.15.22:0/2504339695 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0870003a10 con 0x7f08741053e0 2024-03-24T09:40:33.987 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.119+0000 7f087b9fb700 1 -- 172.21.15.22:0/2504339695 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f087419aa20 con 0x7f08741053e0 2024-03-24T09:40:33.987 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.120+0000 7f086a7fc700 1 -- 172.21.15.22:0/2504339695 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f08700246e0 con 0x7f08741053e0 2024-03-24T09:40:33.987 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.120+0000 7f087b9fb700 1 -- 172.21.15.22:0/2504339695 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f087419aec0 con 0x7f08741053e0 2024-03-24T09:40:33.987 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.120+0000 7f086a7fc700 1 -- 172.21.15.22:0/2504339695 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 11) v1 ==== 45455+0+0 (secure 0 0 0) 0x7f087002d3f0 con 0x7f08741053e0 2024-03-24T09:40:33.987 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.121+0000 7f087b9fb700 1 -- 172.21.15.22:0/2504339695 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0874194420 con 0x7f08741053e0 2024-03-24T09:40:33.987 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.121+0000 7f086a7fc700 1 --2- 172.21.15.22:0/2504339695 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f08600386a0 0x7f086003ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:33.987 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.121+0000 7f086a7fc700 1 -- 172.21.15.22:0/2504339695 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f087004c730 con 0x7f08741053e0 2024-03-24T09:40:33.987 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.121+0000 7f0878f96700 1 --2- 172.21.15.22:0/2504339695 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f08600386a0 0x7f086003ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:33.987 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.121+0000 7f0878f96700 1 --2- 172.21.15.22:0/2504339695 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f08600386a0 0x7f086003ab60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0864006fd0 tx=0x7f0864006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:33.987 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.126+0000 7f086a7fc700 1 -- 172.21.15.22:0/2504339695 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0870010c00 con 0x7f08741053e0 2024-03-24T09:40:33.987 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.334+0000 7f087b9fb700 1 -- 172.21.15.22:0/2504339695 --> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f08740611f0 con 0x7f08600386a0 2024-03-24T09:40:33.987 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.410+0000 7f086a7fc700 1 -- 172.21.15.22:0/2504339695 <== mgr.14164 v2:172.21.15.22:6800/2979399337 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f08740611f0 con 0x7f08600386a0 2024-03-24T09:40:33.988 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.412+0000 7f087b9fb700 1 -- 172.21.15.22:0/2504339695 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f08600386a0 msgr2=0x7f086003ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:33.988 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.412+0000 7f087b9fb700 1 --2- 172.21.15.22:0/2504339695 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f08600386a0 0x7f086003ab60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0864006fd0 tx=0x7f0864006e40 comp rx=0 tx=0).stop 2024-03-24T09:40:33.988 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.412+0000 7f087b9fb700 1 -- 172.21.15.22:0/2504339695 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f08741053e0 msgr2=0x7f087419a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:33.988 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.412+0000 7f087b9fb700 1 --2- 172.21.15.22:0/2504339695 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f08741053e0 0x7f087419a2e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f08700117b0 tx=0x7f0870011890 comp rx=0 tx=0).stop 2024-03-24T09:40:33.988 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.412+0000 7f087b9fb700 1 -- 172.21.15.22:0/2504339695 shutdown_connections 2024-03-24T09:40:33.988 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.412+0000 7f087b9fb700 1 --2- 172.21.15.22:0/2504339695 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f08600386a0 0x7f086003ab60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:33.988 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.412+0000 7f087b9fb700 1 --2- 172.21.15.22:0/2504339695 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f08741053e0 0x7f087419a2e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:33.988 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.412+0000 7f087b9fb700 1 -- 172.21.15.22:0/2504339695 >> 172.21.15.22:0/2504339695 conn(0x7f0874100960 msgr2=0x7f08741018b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:33.988 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.412+0000 7f087b9fb700 1 -- 172.21.15.22:0/2504339695 shutdown_connections 2024-03-24T09:40:33.988 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:33.412+0000 7f087b9fb700 1 -- 172.21.15.22:0/2504339695 wait complete. 2024-03-24T09:40:33.988 INFO:teuthology.orchestra.run.smithi022.stdout:Creating initial admin user... 2024-03-24T09:40:34.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:33 smithi022 ceph-mon[29512]: [24/Mar/2024:09:40:32] ENGINE Serving on https://172.21.15.22:7150 2024-03-24T09:40:34.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:33 smithi022 ceph-mon[29512]: [24/Mar/2024:09:40:32] ENGINE Serving on http://172.21.15.22:8765 2024-03-24T09:40:34.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:33 smithi022 ceph-mon[29512]: [24/Mar/2024:09:40:32] ENGINE Bus STARTED 2024-03-24T09:40:34.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:33 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:34.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:33 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:34.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:33 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:35.281 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:34 smithi022 ceph-mon[29512]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:40:35.281 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:34 smithi022 ceph-mon[29512]: mgrmap e12: smithi022.emgmxz(active, since 3s) 2024-03-24T09:40:35.281 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:34 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:35.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:34 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:36.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:35 smithi022 ceph-mon[29512]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:40:36.281 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$UYl9zU/xVZhKjzOtAeTiyOR9fz2lavwJxkX5d3LViztEQ69G20Xku", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711273234, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-24T09:40:36.281 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.450+0000 7f426e5a0700 1 Processor -- start 2024-03-24T09:40:36.281 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.450+0000 7f426e5a0700 1 -- start start 2024-03-24T09:40:36.281 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.450+0000 7f426e5a0700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4268106f70 0x7f4268109360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:36.281 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.450+0000 7f426e5a0700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42680745b0 con 0x7f4268106f70 2024-03-24T09:40:36.281 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.450+0000 7f4267fff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4268106f70 0x7f4268109360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:36.281 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.450+0000 7f4267fff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4268106f70 0x7f4268109360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:34722/0 (socket says 172.21.15.22:34722) 2024-03-24T09:40:36.281 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.451+0000 7f4267fff700 1 -- 172.21.15.22:0/2850254574 learned_addr learned my addr 172.21.15.22:0/2850254574 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:36.282 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.451+0000 7f4267fff700 1 -- 172.21.15.22:0/2850254574 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42681098a0 con 0x7f4268106f70 2024-03-24T09:40:36.282 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.451+0000 7f4267fff700 1 --2- 172.21.15.22:0/2850254574 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4268106f70 0x7f4268109360 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f425c0098a0 tx=0x7f425c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca4c42097fd9533e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:36.282 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.451+0000 7f4266ffd700 1 -- 172.21.15.22:0/2850254574 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f425c004180 con 0x7f4268106f70 2024-03-24T09:40:36.282 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.451+0000 7f4266ffd700 1 -- 172.21.15.22:0/2850254574 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f425c0042e0 con 0x7f4268106f70 2024-03-24T09:40:36.282 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.451+0000 7f4266ffd700 1 -- 172.21.15.22:0/2850254574 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f425c011b10 con 0x7f4268106f70 2024-03-24T09:40:36.282 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.452+0000 7f426e5a0700 1 -- 172.21.15.22:0/2850254574 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4268106f70 msgr2=0x7f4268109360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:36.282 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.452+0000 7f426e5a0700 1 --2- 172.21.15.22:0/2850254574 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4268106f70 0x7f4268109360 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f425c0098a0 tx=0x7f425c009bb0 comp rx=0 tx=0).stop 2024-03-24T09:40:36.282 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.452+0000 7f426e5a0700 1 -- 172.21.15.22:0/2850254574 shutdown_connections 2024-03-24T09:40:36.282 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.452+0000 7f426e5a0700 1 --2- 172.21.15.22:0/2850254574 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4268106f70 0x7f4268109360 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:36.283 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.452+0000 7f426e5a0700 1 -- 172.21.15.22:0/2850254574 >> 172.21.15.22:0/2850254574 conn(0x7f42681009c0 msgr2=0x7f4268102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:36.283 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.452+0000 7f426e5a0700 1 -- 172.21.15.22:0/2850254574 shutdown_connections 2024-03-24T09:40:36.283 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.452+0000 7f426e5a0700 1 -- 172.21.15.22:0/2850254574 wait complete. 2024-03-24T09:40:36.283 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.453+0000 7f426e5a0700 1 Processor -- start 2024-03-24T09:40:36.283 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.453+0000 7f426e5a0700 1 -- start start 2024-03-24T09:40:36.283 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.453+0000 7f426e5a0700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4268106f70 0x7f426819a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:36.283 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.453+0000 7f426e5a0700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426819a760 con 0x7f4268106f70 2024-03-24T09:40:36.283 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.453+0000 7f4267fff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4268106f70 0x7f426819a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:36.283 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.453+0000 7f4267fff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4268106f70 0x7f426819a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:34734/0 (socket says 172.21.15.22:34734) 2024-03-24T09:40:36.283 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.453+0000 7f4267fff700 1 -- 172.21.15.22:0/3019962873 learned_addr learned my addr 172.21.15.22:0/3019962873 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:36.284 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.453+0000 7f4267fff700 1 -- 172.21.15.22:0/3019962873 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f425c009550 con 0x7f4268106f70 2024-03-24T09:40:36.284 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.454+0000 7f4267fff700 1 --2- 172.21.15.22:0/3019962873 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4268106f70 0x7f426819a220 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f425c0117b0 tx=0x7f425c011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:36.284 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.454+0000 7f42657fa700 1 -- 172.21.15.22:0/3019962873 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f425c0038b0 con 0x7f4268106f70 2024-03-24T09:40:36.284 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.454+0000 7f42657fa700 1 -- 172.21.15.22:0/3019962873 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f425c003a10 con 0x7f4268106f70 2024-03-24T09:40:36.284 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.454+0000 7f42657fa700 1 -- 172.21.15.22:0/3019962873 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f425c024780 con 0x7f4268106f70 2024-03-24T09:40:36.284 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.454+0000 7f426e5a0700 1 -- 172.21.15.22:0/3019962873 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f426819a960 con 0x7f4268106f70 2024-03-24T09:40:36.284 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.454+0000 7f426e5a0700 1 -- 172.21.15.22:0/3019962873 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f426819ae00 con 0x7f4268106f70 2024-03-24T09:40:36.284 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.455+0000 7f42657fa700 1 -- 172.21.15.22:0/3019962873 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 12) v1 ==== 45561+0+0 (secure 0 0 0) 0x7f425c0248e0 con 0x7f4268106f70 2024-03-24T09:40:36.284 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.455+0000 7f42657fa700 1 --2- 172.21.15.22:0/3019962873 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f42500387c0 0x7f425003ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:36.284 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.455+0000 7f426e5a0700 1 -- 172.21.15.22:0/3019962873 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f426810a630 con 0x7f4268106f70 2024-03-24T09:40:36.285 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.455+0000 7f42657fa700 1 -- 172.21.15.22:0/3019962873 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f425c04c8d0 con 0x7f4268106f70 2024-03-24T09:40:36.285 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.455+0000 7f42677fe700 1 --2- 172.21.15.22:0/3019962873 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f42500387c0 0x7f425003ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:36.285 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.456+0000 7f42677fe700 1 --2- 172.21.15.22:0/3019962873 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f42500387c0 0x7f425003ac80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4258006fd0 tx=0x7f4258006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:36.285 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.460+0000 7f42657fa700 1 -- 172.21.15.22:0/3019962873 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f425c048020 con 0x7f4268106f70 2024-03-24T09:40:36.285 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.686+0000 7f426e5a0700 1 -- 172.21.15.22:0/3019962873 --> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f426819db70 con 0x7f42500387c0 2024-03-24T09:40:36.285 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.933+0000 7f42657fa700 1 -- 172.21.15.22:0/3019962873 <== mgr.14164 v2:172.21.15.22:6800/2979399337 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f426819db70 con 0x7f42500387c0 2024-03-24T09:40:36.285 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.935+0000 7f426e5a0700 1 -- 172.21.15.22:0/3019962873 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f42500387c0 msgr2=0x7f425003ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:36.285 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.935+0000 7f426e5a0700 1 --2- 172.21.15.22:0/3019962873 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f42500387c0 0x7f425003ac80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4258006fd0 tx=0x7f4258006e40 comp rx=0 tx=0).stop 2024-03-24T09:40:36.285 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.936+0000 7f426e5a0700 1 -- 172.21.15.22:0/3019962873 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4268106f70 msgr2=0x7f426819a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:36.285 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.936+0000 7f426e5a0700 1 --2- 172.21.15.22:0/3019962873 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4268106f70 0x7f426819a220 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f425c0117b0 tx=0x7f425c011890 comp rx=0 tx=0).stop 2024-03-24T09:40:36.286 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.936+0000 7f426e5a0700 1 -- 172.21.15.22:0/3019962873 shutdown_connections 2024-03-24T09:40:36.286 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.936+0000 7f426e5a0700 1 --2- 172.21.15.22:0/3019962873 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f42500387c0 0x7f425003ac80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:36.286 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.936+0000 7f426e5a0700 1 --2- 172.21.15.22:0/3019962873 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4268106f70 0x7f426819a220 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:36.286 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.936+0000 7f426e5a0700 1 -- 172.21.15.22:0/3019962873 >> 172.21.15.22:0/3019962873 conn(0x7f42681009c0 msgr2=0x7f4268101550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:36.286 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.936+0000 7f426e5a0700 1 -- 172.21.15.22:0/3019962873 shutdown_connections 2024-03-24T09:40:36.286 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:34.936+0000 7f426e5a0700 1 -- 172.21.15.22:0/3019962873 wait complete. 2024-03-24T09:40:36.286 INFO:teuthology.orchestra.run.smithi022.stdout:Fetching dashboard port number... 2024-03-24T09:40:37.237 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:36 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/3023535052' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-24T09:40:37.489 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stdout 8443 2024-03-24T09:40:37.489 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.726+0000 7f6796647700 1 Processor -- start 2024-03-24T09:40:37.489 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.726+0000 7f6796647700 1 -- start start 2024-03-24T09:40:37.490 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.726+0000 7f6796647700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6790106fa0 0x7f6790109390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:37.490 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.726+0000 7f6796647700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6790079e50 con 0x7f6790106fa0 2024-03-24T09:40:37.490 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.727+0000 7f678ffff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6790106fa0 0x7f6790109390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:37.490 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.727+0000 7f678ffff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6790106fa0 0x7f6790109390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:34750/0 (socket says 172.21.15.22:34750) 2024-03-24T09:40:37.490 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.727+0000 7f678ffff700 1 -- 172.21.15.22:0/2327288787 learned_addr learned my addr 172.21.15.22:0/2327288787 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:37.490 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.727+0000 7f678ffff700 1 -- 172.21.15.22:0/2327288787 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67901098d0 con 0x7f6790106fa0 2024-03-24T09:40:37.490 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.727+0000 7f678ffff700 1 --2- 172.21.15.22:0/2327288787 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6790106fa0 0x7f6790109390 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f67800098a0 tx=0x7f6780009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa3b89e3ccc95ab8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:37.490 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.727+0000 7f678effd700 1 -- 172.21.15.22:0/2327288787 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6780004180 con 0x7f6790106fa0 2024-03-24T09:40:37.490 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.727+0000 7f678effd700 1 -- 172.21.15.22:0/2327288787 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f67800042e0 con 0x7f6790106fa0 2024-03-24T09:40:37.490 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.727+0000 7f678effd700 1 -- 172.21.15.22:0/2327288787 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6780011b10 con 0x7f6790106fa0 2024-03-24T09:40:37.491 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.728+0000 7f6796647700 1 -- 172.21.15.22:0/2327288787 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6790106fa0 msgr2=0x7f6790109390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:37.491 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.728+0000 7f6796647700 1 --2- 172.21.15.22:0/2327288787 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6790106fa0 0x7f6790109390 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f67800098a0 tx=0x7f6780009bb0 comp rx=0 tx=0).stop 2024-03-24T09:40:37.491 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.728+0000 7f6796647700 1 -- 172.21.15.22:0/2327288787 shutdown_connections 2024-03-24T09:40:37.491 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.728+0000 7f6796647700 1 --2- 172.21.15.22:0/2327288787 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6790106fa0 0x7f6790109390 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:37.491 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.728+0000 7f6796647700 1 -- 172.21.15.22:0/2327288787 >> 172.21.15.22:0/2327288787 conn(0x7f67901009d0 msgr2=0x7f6790102df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:37.491 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.728+0000 7f6796647700 1 -- 172.21.15.22:0/2327288787 shutdown_connections 2024-03-24T09:40:37.491 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.728+0000 7f6796647700 1 -- 172.21.15.22:0/2327288787 wait complete. 2024-03-24T09:40:37.491 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.729+0000 7f6796647700 1 Processor -- start 2024-03-24T09:40:37.491 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.729+0000 7f6796647700 1 -- start start 2024-03-24T09:40:37.491 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.729+0000 7f6796647700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6790106fa0 0x7f679019a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:37.492 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.729+0000 7f6796647700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679019a780 con 0x7f6790106fa0 2024-03-24T09:40:37.492 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.729+0000 7f678ffff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6790106fa0 0x7f679019a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:37.492 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.729+0000 7f678ffff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6790106fa0 0x7f679019a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:34766/0 (socket says 172.21.15.22:34766) 2024-03-24T09:40:37.492 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.729+0000 7f678ffff700 1 -- 172.21.15.22:0/3023535052 learned_addr learned my addr 172.21.15.22:0/3023535052 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:37.492 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.729+0000 7f678ffff700 1 -- 172.21.15.22:0/3023535052 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6780009550 con 0x7f6790106fa0 2024-03-24T09:40:37.492 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.730+0000 7f678ffff700 1 --2- 172.21.15.22:0/3023535052 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6790106fa0 0x7f679019a240 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f67800117b0 tx=0x7f6780011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:37.492 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.730+0000 7f678d7fa700 1 -- 172.21.15.22:0/3023535052 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67800038b0 con 0x7f6790106fa0 2024-03-24T09:40:37.492 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.730+0000 7f6796647700 1 -- 172.21.15.22:0/3023535052 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f679019a980 con 0x7f6790106fa0 2024-03-24T09:40:37.492 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.730+0000 7f678d7fa700 1 -- 172.21.15.22:0/3023535052 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6780003a10 con 0x7f6790106fa0 2024-03-24T09:40:37.492 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.730+0000 7f678d7fa700 1 -- 172.21.15.22:0/3023535052 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6780024780 con 0x7f6790106fa0 2024-03-24T09:40:37.493 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.730+0000 7f6796647700 1 -- 172.21.15.22:0/3023535052 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f679019ae20 con 0x7f6790106fa0 2024-03-24T09:40:37.493 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.731+0000 7f678d7fa700 1 -- 172.21.15.22:0/3023535052 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 12) v1 ==== 45561+0+0 (secure 0 0 0) 0x7f678001a430 con 0x7f6790106fa0 2024-03-24T09:40:37.493 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.731+0000 7f6796647700 1 -- 172.21.15.22:0/3023535052 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f679010a660 con 0x7f6790106fa0 2024-03-24T09:40:37.493 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.731+0000 7f678d7fa700 1 --2- 172.21.15.22:0/3023535052 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f6778038770 0x7f677803ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:37.493 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.731+0000 7f678d7fa700 1 -- 172.21.15.22:0/3023535052 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f678004c8b0 con 0x7f6790106fa0 2024-03-24T09:40:37.493 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.731+0000 7f678f7fe700 1 --2- 172.21.15.22:0/3023535052 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f6778038770 0x7f677803ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:37.493 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.731+0000 7f678f7fe700 1 --2- 172.21.15.22:0/3023535052 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f6778038770 0x7f677803ac30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6784006fd0 tx=0x7f6784006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:37.493 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.736+0000 7f678d7fa700 1 -- 172.21.15.22:0/3023535052 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6780018350 con 0x7f6790106fa0 2024-03-24T09:40:37.493 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.938+0000 7f6796647700 1 -- 172.21.15.22:0/3023535052 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f679010a850 con 0x7f6790106fa0 2024-03-24T09:40:37.493 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.939+0000 7f678d7fa700 1 -- 172.21.15.22:0/3023535052 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f6780018350 con 0x7f6790106fa0 2024-03-24T09:40:37.494 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.941+0000 7f6796647700 1 -- 172.21.15.22:0/3023535052 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f6778038770 msgr2=0x7f677803ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:37.494 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.941+0000 7f6796647700 1 --2- 172.21.15.22:0/3023535052 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f6778038770 0x7f677803ac30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6784006fd0 tx=0x7f6784006e40 comp rx=0 tx=0).stop 2024-03-24T09:40:37.494 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.941+0000 7f6796647700 1 -- 172.21.15.22:0/3023535052 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6790106fa0 msgr2=0x7f679019a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:37.494 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.941+0000 7f6796647700 1 --2- 172.21.15.22:0/3023535052 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6790106fa0 0x7f679019a240 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f67800117b0 tx=0x7f6780011890 comp rx=0 tx=0).stop 2024-03-24T09:40:37.494 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.941+0000 7f6796647700 1 -- 172.21.15.22:0/3023535052 shutdown_connections 2024-03-24T09:40:37.494 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.941+0000 7f6796647700 1 --2- 172.21.15.22:0/3023535052 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f6778038770 0x7f677803ac30 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:37.494 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.941+0000 7f6796647700 1 --2- 172.21.15.22:0/3023535052 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6790106fa0 0x7f679019a240 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:37.494 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.941+0000 7f6796647700 1 -- 172.21.15.22:0/3023535052 >> 172.21.15.22:0/3023535052 conn(0x7f67901009d0 msgr2=0x7f6790101480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:37.494 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.941+0000 7f6796647700 1 -- 172.21.15.22:0/3023535052 shutdown_connections 2024-03-24T09:40:37.494 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:36.941+0000 7f6796647700 1 -- 172.21.15.22:0/3023535052 wait complete. 2024-03-24T09:40:37.495 INFO:teuthology.orchestra.run.smithi022.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T09:40:37.495 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stdout disabled 2024-03-24T09:40:37.501 INFO:teuthology.orchestra.run.smithi022.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T09:40:37.501 INFO:teuthology.orchestra.run.smithi022.stdout:systemctl: stdout inactive 2024-03-24T09:40:37.501 INFO:teuthology.orchestra.run.smithi022.stdout:firewalld.service is not enabled 2024-03-24T09:40:37.501 INFO:teuthology.orchestra.run.smithi022.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-24T09:40:37.502 INFO:teuthology.orchestra.run.smithi022.stdout:Ceph Dashboard is now available at: 2024-03-24T09:40:37.502 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:40:37.502 INFO:teuthology.orchestra.run.smithi022.stdout: URL: https://smithi022.front.sepia.ceph.com:8443/ 2024-03-24T09:40:37.503 INFO:teuthology.orchestra.run.smithi022.stdout: User: admin 2024-03-24T09:40:37.503 INFO:teuthology.orchestra.run.smithi022.stdout: Password: 67584tcf5h 2024-03-24T09:40:37.503 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:40:37.503 INFO:teuthology.orchestra.run.smithi022.stdout:Saving cluster configuration to /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/config directory 2024-03-24T09:40:37.503 INFO:teuthology.orchestra.run.smithi022.stdout:Enabling autotune for osd_memory_target 2024-03-24T09:40:38.645 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.935+0000 7f3cd7349700 1 Processor -- start 2024-03-24T09:40:38.645 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.935+0000 7f3cd7349700 1 -- start start 2024-03-24T09:40:38.645 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.936+0000 7f3cd7349700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3cd0108720 0x7f3cd0108b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:38.646 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.936+0000 7f3cd7349700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd0109110 con 0x7f3cd0108720 2024-03-24T09:40:38.646 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.936+0000 7f3cd50e5700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3cd0108720 0x7f3cd0108b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:38.646 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.936+0000 7f3cd50e5700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3cd0108720 0x7f3cd0108b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:34768/0 (socket says 172.21.15.22:34768) 2024-03-24T09:40:38.646 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.936+0000 7f3cd50e5700 1 -- 172.21.15.22:0/4227627998 learned_addr learned my addr 172.21.15.22:0/4227627998 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:38.646 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.936+0000 7f3cd50e5700 1 -- 172.21.15.22:0/4227627998 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd01098d0 con 0x7f3cd0108720 2024-03-24T09:40:38.646 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.936+0000 7f3cd50e5700 1 --2- 172.21.15.22:0/4227627998 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3cd0108720 0x7f3cd0108b40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc0098a0 tx=0x7f3ccc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=767cb3dd9018184f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:38.646 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.936+0000 7f3cc7fff700 1 -- 172.21.15.22:0/4227627998 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ccc004180 con 0x7f3cd0108720 2024-03-24T09:40:38.646 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.936+0000 7f3cc7fff700 1 -- 172.21.15.22:0/4227627998 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3ccc0042e0 con 0x7f3cd0108720 2024-03-24T09:40:38.647 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.936+0000 7f3cc7fff700 1 -- 172.21.15.22:0/4227627998 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ccc011b10 con 0x7f3cd0108720 2024-03-24T09:40:38.647 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.937+0000 7f3cd7349700 1 -- 172.21.15.22:0/4227627998 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3cd0108720 msgr2=0x7f3cd0108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:38.647 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.937+0000 7f3cd7349700 1 --2- 172.21.15.22:0/4227627998 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3cd0108720 0x7f3cd0108b40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc0098a0 tx=0x7f3ccc009bb0 comp rx=0 tx=0).stop 2024-03-24T09:40:38.647 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.937+0000 7f3cd7349700 1 -- 172.21.15.22:0/4227627998 shutdown_connections 2024-03-24T09:40:38.647 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.937+0000 7f3cd7349700 1 --2- 172.21.15.22:0/4227627998 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3cd0108720 0x7f3cd0108b40 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:38.647 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.937+0000 7f3cd7349700 1 -- 172.21.15.22:0/4227627998 >> 172.21.15.22:0/4227627998 conn(0x7f3cd007bdf0 msgr2=0x7f3cd01064b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:38.647 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.937+0000 7f3cd7349700 1 -- 172.21.15.22:0/4227627998 shutdown_connections 2024-03-24T09:40:38.647 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.937+0000 7f3cd7349700 1 -- 172.21.15.22:0/4227627998 wait complete. 2024-03-24T09:40:38.647 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.938+0000 7f3cd7349700 1 Processor -- start 2024-03-24T09:40:38.647 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.938+0000 7f3cd7349700 1 -- start start 2024-03-24T09:40:38.648 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.938+0000 7f3cd7349700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3cd0108720 0x7f3cd019c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:38.648 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.938+0000 7f3cd7349700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd019c950 con 0x7f3cd0108720 2024-03-24T09:40:38.648 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.938+0000 7f3cd50e5700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3cd0108720 0x7f3cd019c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:38.648 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.938+0000 7f3cd50e5700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3cd0108720 0x7f3cd019c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:34774/0 (socket says 172.21.15.22:34774) 2024-03-24T09:40:38.648 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.938+0000 7f3cd50e5700 1 -- 172.21.15.22:0/2764175229 learned_addr learned my addr 172.21.15.22:0/2764175229 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:38.648 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.938+0000 7f3cd50e5700 1 -- 172.21.15.22:0/2764175229 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ccc009550 con 0x7f3cd0108720 2024-03-24T09:40:38.648 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.938+0000 7f3cd50e5700 1 --2- 172.21.15.22:0/2764175229 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3cd0108720 0x7f3cd019c410 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc009fb0 tx=0x7f3ccc011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:38.648 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.938+0000 7f3cc67fc700 1 -- 172.21.15.22:0/2764175229 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ccc0038b0 con 0x7f3cd0108720 2024-03-24T09:40:38.648 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.939+0000 7f3cc67fc700 1 -- 172.21.15.22:0/2764175229 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3ccc003a10 con 0x7f3cd0108720 2024-03-24T09:40:38.649 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.939+0000 7f3cd7349700 1 -- 172.21.15.22:0/2764175229 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cd019cb50 con 0x7f3cd0108720 2024-03-24T09:40:38.649 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.939+0000 7f3cc67fc700 1 -- 172.21.15.22:0/2764175229 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ccc019770 con 0x7f3cd0108720 2024-03-24T09:40:38.649 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.939+0000 7f3cd7349700 1 -- 172.21.15.22:0/2764175229 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cd019cff0 con 0x7f3cd0108720 2024-03-24T09:40:38.649 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.940+0000 7f3cd7349700 1 -- 172.21.15.22:0/2764175229 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cd010cb00 con 0x7f3cd0108720 2024-03-24T09:40:38.649 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.940+0000 7f3cc67fc700 1 -- 172.21.15.22:0/2764175229 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 12) v1 ==== 45561+0+0 (secure 0 0 0) 0x7f3ccc0198d0 con 0x7f3cd0108720 2024-03-24T09:40:38.649 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.940+0000 7f3cc67fc700 1 --2- 172.21.15.22:0/2764175229 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f3cbc038780 0x7f3cbc03ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:38.649 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.940+0000 7f3cc67fc700 1 -- 172.21.15.22:0/2764175229 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3ccc04ca50 con 0x7f3cd0108720 2024-03-24T09:40:38.652 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.940+0000 7f3cd48e4700 1 --2- 172.21.15.22:0/2764175229 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f3cbc038780 0x7f3cbc03ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:38.652 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.940+0000 7f3cd48e4700 1 --2- 172.21.15.22:0/2764175229 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f3cbc038780 0x7f3cbc03ac40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3cc0006fd0 tx=0x7f3cc0006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:38.653 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.945+0000 7f3cc67fc700 1 -- 172.21.15.22:0/2764175229 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3ccc0224f0 con 0x7f3cd0108720 2024-03-24T09:40:38.653 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:37.955+0000 7f3cc67fc700 1 -- 172.21.15.22:0/2764175229 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mgrmap(e 13) v1 ==== 45604+0+0 (secure 0 0 0) 0x7f3ccc01ac10 con 0x7f3cd0108720 2024-03-24T09:40:38.653 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:38.145+0000 7f3cd7349700 1 -- 172.21.15.22:0/2764175229 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f3cd019fca0 con 0x7f3cd0108720 2024-03-24T09:40:38.653 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:38.145+0000 7f3cc67fc700 1 -- 172.21.15.22:0/2764175229 <== mon.0 v2:172.21.15.22:3300/0 8 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f3ccc022e60 con 0x7f3cd0108720 2024-03-24T09:40:38.653 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:38.147+0000 7f3cd7349700 1 -- 172.21.15.22:0/2764175229 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f3cbc038780 msgr2=0x7f3cbc03ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:38.653 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:38.147+0000 7f3cd7349700 1 --2- 172.21.15.22:0/2764175229 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f3cbc038780 0x7f3cbc03ac40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3cc0006fd0 tx=0x7f3cc0006e40 comp rx=0 tx=0).stop 2024-03-24T09:40:38.653 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:38.147+0000 7f3cd7349700 1 -- 172.21.15.22:0/2764175229 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3cd0108720 msgr2=0x7f3cd019c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:38.653 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:38.147+0000 7f3cd7349700 1 --2- 172.21.15.22:0/2764175229 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3cd0108720 0x7f3cd019c410 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc009fb0 tx=0x7f3ccc011890 comp rx=0 tx=0).stop 2024-03-24T09:40:38.653 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:38.147+0000 7f3cd7349700 1 -- 172.21.15.22:0/2764175229 shutdown_connections 2024-03-24T09:40:38.653 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:38.147+0000 7f3cd7349700 1 --2- 172.21.15.22:0/2764175229 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f3cbc038780 0x7f3cbc03ac40 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:38.654 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:38.147+0000 7f3cd7349700 1 --2- 172.21.15.22:0/2764175229 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3cd0108720 0x7f3cd019c410 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:38.654 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:38.147+0000 7f3cd7349700 1 -- 172.21.15.22:0/2764175229 >> 172.21.15.22:0/2764175229 conn(0x7f3cd007bdf0 msgr2=0x7f3cd0105ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:38.654 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:38.148+0000 7f3cd7349700 1 -- 172.21.15.22:0/2764175229 shutdown_connections 2024-03-24T09:40:38.654 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:38.148+0000 7f3cd7349700 1 -- 172.21.15.22:0/2764175229 wait complete. 2024-03-24T09:40:39.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:38 smithi022 ceph-mon[29512]: mgrmap e13: smithi022.emgmxz(active, since 7s) 2024-03-24T09:40:39.977 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.080+0000 7f20a9edf700 1 Processor -- start 2024-03-24T09:40:39.977 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.080+0000 7f20a9edf700 1 -- start start 2024-03-24T09:40:39.977 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.080+0000 7f20a9edf700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f20a4106fd0 0x7f20a41093c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:39.977 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.080+0000 7f20a9edf700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a4074550 con 0x7f20a4106fd0 2024-03-24T09:40:39.977 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.081+0000 7f20a37fe700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f20a4106fd0 0x7f20a41093c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:39.978 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.081+0000 7f20a37fe700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f20a4106fd0 0x7f20a41093c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60412/0 (socket says 172.21.15.22:60412) 2024-03-24T09:40:39.978 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.081+0000 7f20a37fe700 1 -- 172.21.15.22:0/2521336230 learned_addr learned my addr 172.21.15.22:0/2521336230 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:39.978 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.081+0000 7f20a37fe700 1 -- 172.21.15.22:0/2521336230 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a4074690 con 0x7f20a4106fd0 2024-03-24T09:40:39.978 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.081+0000 7f20a37fe700 1 --2- 172.21.15.22:0/2521336230 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f20a4106fd0 0x7f20a41093c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f209c0098a0 tx=0x7f209c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60c5ef6042b6fd42 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:39.978 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.081+0000 7f20a27fc700 1 -- 172.21.15.22:0/2521336230 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f209c004180 con 0x7f20a4106fd0 2024-03-24T09:40:39.978 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.081+0000 7f20a27fc700 1 -- 172.21.15.22:0/2521336230 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f209c0042e0 con 0x7f20a4106fd0 2024-03-24T09:40:39.978 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.081+0000 7f20a27fc700 1 -- 172.21.15.22:0/2521336230 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f209c011b10 con 0x7f20a4106fd0 2024-03-24T09:40:39.978 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.082+0000 7f20a9edf700 1 -- 172.21.15.22:0/2521336230 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f20a4106fd0 msgr2=0x7f20a41093c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:39.978 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.082+0000 7f20a9edf700 1 --2- 172.21.15.22:0/2521336230 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f20a4106fd0 0x7f20a41093c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f209c0098a0 tx=0x7f209c009bb0 comp rx=0 tx=0).stop 2024-03-24T09:40:39.979 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.082+0000 7f20a9edf700 1 -- 172.21.15.22:0/2521336230 shutdown_connections 2024-03-24T09:40:39.979 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.082+0000 7f20a9edf700 1 --2- 172.21.15.22:0/2521336230 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f20a4106fd0 0x7f20a41093c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:39.979 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.082+0000 7f20a9edf700 1 -- 172.21.15.22:0/2521336230 >> 172.21.15.22:0/2521336230 conn(0x7f20a41009c0 msgr2=0x7f20a4102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:39.979 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.082+0000 7f20a9edf700 1 -- 172.21.15.22:0/2521336230 shutdown_connections 2024-03-24T09:40:39.979 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.082+0000 7f20a9edf700 1 -- 172.21.15.22:0/2521336230 wait complete. 2024-03-24T09:40:39.979 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.082+0000 7f20a9edf700 1 Processor -- start 2024-03-24T09:40:39.979 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.083+0000 7f20a9edf700 1 -- start start 2024-03-24T09:40:39.979 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.083+0000 7f20a9edf700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f20a4106fd0 0x7f20a419a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:39.979 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.083+0000 7f20a9edf700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a419a750 con 0x7f20a4106fd0 2024-03-24T09:40:39.980 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.083+0000 7f20a37fe700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f20a4106fd0 0x7f20a419a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:39.980 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.083+0000 7f20a37fe700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f20a4106fd0 0x7f20a419a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60420/0 (socket says 172.21.15.22:60420) 2024-03-24T09:40:39.980 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.083+0000 7f20a37fe700 1 -- 172.21.15.22:0/1831129017 learned_addr learned my addr 172.21.15.22:0/1831129017 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:39.980 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.083+0000 7f20a37fe700 1 -- 172.21.15.22:0/1831129017 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f209c009550 con 0x7f20a4106fd0 2024-03-24T09:40:39.980 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.083+0000 7f20a37fe700 1 --2- 172.21.15.22:0/1831129017 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f20a4106fd0 0x7f20a419a210 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f209c009fb0 tx=0x7f209c011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:39.980 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.083+0000 7f20a0ff9700 1 -- 172.21.15.22:0/1831129017 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f209c0038b0 con 0x7f20a4106fd0 2024-03-24T09:40:39.980 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.084+0000 7f20a0ff9700 1 -- 172.21.15.22:0/1831129017 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f209c003a10 con 0x7f20a4106fd0 2024-03-24T09:40:39.980 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.084+0000 7f20a9edf700 1 -- 172.21.15.22:0/1831129017 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20a419a950 con 0x7f20a4106fd0 2024-03-24T09:40:39.980 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.084+0000 7f20a0ff9700 1 -- 172.21.15.22:0/1831129017 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f209c019750 con 0x7f20a4106fd0 2024-03-24T09:40:39.980 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.084+0000 7f20a9edf700 1 -- 172.21.15.22:0/1831129017 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20a419adf0 con 0x7f20a4106fd0 2024-03-24T09:40:39.981 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.084+0000 7f20a0ff9700 1 -- 172.21.15.22:0/1831129017 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 13) v1 ==== 45604+0+0 (secure 0 0 0) 0x7f209c0198b0 con 0x7f20a4106fd0 2024-03-24T09:40:39.981 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.085+0000 7f20a0ff9700 1 --2- 172.21.15.22:0/1831129017 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f2090038820 0x7f209003ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:39.981 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.085+0000 7f20a0ff9700 1 -- 172.21.15.22:0/1831129017 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f209c04ca80 con 0x7f20a4106fd0 2024-03-24T09:40:39.981 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.085+0000 7f20a9edf700 1 -- 172.21.15.22:0/1831129017 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20a410a620 con 0x7f20a4106fd0 2024-03-24T09:40:39.981 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.085+0000 7f20a2ffd700 1 --2- 172.21.15.22:0/1831129017 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f2090038820 0x7f209003ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:39.981 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.085+0000 7f20a2ffd700 1 --2- 172.21.15.22:0/1831129017 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f2090038820 0x7f209003ace0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f208c006fd0 tx=0x7f208c006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:39.981 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.090+0000 7f20a0ff9700 1 -- 172.21.15.22:0/1831129017 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f209c019c00 con 0x7f20a4106fd0 2024-03-24T09:40:39.981 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.403+0000 7f20a9edf700 1 -- 172.21.15.22:0/1831129017 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f20a419daa0 con 0x7f20a4106fd0 2024-03-24T09:40:39.981 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.405+0000 7f20a0ff9700 1 -- 172.21.15.22:0/1831129017 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v34)=0 set mgr/dashboard/cluster/status v34) v1 ==== 153+0+0 (secure 0 0 0) 0x7f209c01ac10 con 0x7f20a4106fd0 2024-03-24T09:40:39.982 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.407+0000 7f20a9edf700 1 -- 172.21.15.22:0/1831129017 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f2090038820 msgr2=0x7f209003ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:39.982 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.407+0000 7f20a9edf700 1 --2- 172.21.15.22:0/1831129017 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f2090038820 0x7f209003ace0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f208c006fd0 tx=0x7f208c006e40 comp rx=0 tx=0).stop 2024-03-24T09:40:39.982 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.407+0000 7f20a9edf700 1 -- 172.21.15.22:0/1831129017 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f20a4106fd0 msgr2=0x7f20a419a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:39.982 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.407+0000 7f20a9edf700 1 --2- 172.21.15.22:0/1831129017 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f20a4106fd0 0x7f20a419a210 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f209c009fb0 tx=0x7f209c011890 comp rx=0 tx=0).stop 2024-03-24T09:40:39.982 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.407+0000 7f20a9edf700 1 -- 172.21.15.22:0/1831129017 shutdown_connections 2024-03-24T09:40:39.982 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.407+0000 7f20a9edf700 1 --2- 172.21.15.22:0/1831129017 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f2090038820 0x7f209003ace0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:39.982 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.407+0000 7f20a9edf700 1 --2- 172.21.15.22:0/1831129017 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f20a4106fd0 0x7f20a419a210 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:39.982 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.407+0000 7f20a9edf700 1 -- 172.21.15.22:0/1831129017 >> 172.21.15.22:0/1831129017 conn(0x7f20a41009c0 msgr2=0x7f20a4101520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:39.982 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.408+0000 7f20a9edf700 1 -- 172.21.15.22:0/1831129017 shutdown_connections 2024-03-24T09:40:39.983 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr 2024-03-24T09:40:39.408+0000 7f20a9edf700 1 -- 172.21.15.22:0/1831129017 wait complete. 2024-03-24T09:40:39.983 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-24T09:40:39.983 INFO:teuthology.orchestra.run.smithi022.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-24T09:40:39.983 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:40:39.983 INFO:teuthology.orchestra.run.smithi022.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-24T09:40:39.983 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:40:39.983 INFO:teuthology.orchestra.run.smithi022.stdout:Or, if you are only running a single cluster on this host: 2024-03-24T09:40:39.983 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:40:39.983 INFO:teuthology.orchestra.run.smithi022.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-24T09:40:39.983 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:40:39.984 INFO:teuthology.orchestra.run.smithi022.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-24T09:40:39.984 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:40:39.984 INFO:teuthology.orchestra.run.smithi022.stdout: ceph telemetry on 2024-03-24T09:40:39.984 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:40:39.984 INFO:teuthology.orchestra.run.smithi022.stdout:For more information see: 2024-03-24T09:40:39.984 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:40:39.984 INFO:teuthology.orchestra.run.smithi022.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-24T09:40:39.984 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:40:39.984 INFO:teuthology.orchestra.run.smithi022.stdout:Bootstrap complete. 2024-03-24T09:40:40.012 INFO:tasks.cephadm:Fetching config... 2024-03-24T09:40:40.013 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T09:40:40.013 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-24T09:40:40.030 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-24T09:40:40.030 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T09:40:40.030 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-24T09:40:40.085 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-24T09:40:40.085 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T09:40:40.085 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/keyring of=/dev/stdout 2024-03-24T09:40:40.153 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-24T09:40:40.153 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T09:40:40.153 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-24T09:40:40.207 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-24T09:40:40.208 DEBUG:teuthology.orchestra.run.smithi022:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDNcYlsM3foEXSl36/mFg/fy7bIaYscKfx79D+1lQ6b5DszL+AkkZZKMsTkQZjpQiKFn4gIDjgomzNLJxwJmslxl1eZpZUjHK9rHW4QACShEQdsU8Jn3rnhw9iZv6UdVEqVw/KNtO6jSdhsJaobCp8NjwNxB5sBqnuvTmSqClvHFqcaC+/cEpGrZQnjkni+kT8FlL+Mc7JrZRtRrpF/u3z2LpHUoyOcty88EyRn5vzKu4pqAuuzo9iD6of0NNhPTuI19EpjWVemSyTBi0PogQxw15sChCebC5UBJhqddUxhZ1T6H5RjCY3vl9edvGtV9YJ+rynjqfBUsBb3B5qS+WgI95IJ3dY7h4HVhKFX6s76GZ0AvfpJYngp9aZu36p1/IsW70UjwaIE7WO+HiyUGbPxoFq1xGGRbnKSALTbyzcR33LvJltGjPTuKs9eZTMUCSD0bPw3UAMfQrZ06DcdWYNZB4OYeofArGE9n4YqIkHEfV1AQgHH9SavVAvPA4LHaM= ceph-45356448-e9c2-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T09:40:40.292 INFO:teuthology.orchestra.run.smithi022.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDNcYlsM3foEXSl36/mFg/fy7bIaYscKfx79D+1lQ6b5DszL+AkkZZKMsTkQZjpQiKFn4gIDjgomzNLJxwJmslxl1eZpZUjHK9rHW4QACShEQdsU8Jn3rnhw9iZv6UdVEqVw/KNtO6jSdhsJaobCp8NjwNxB5sBqnuvTmSqClvHFqcaC+/cEpGrZQnjkni+kT8FlL+Mc7JrZRtRrpF/u3z2LpHUoyOcty88EyRn5vzKu4pqAuuzo9iD6of0NNhPTuI19EpjWVemSyTBi0PogQxw15sChCebC5UBJhqddUxhZ1T6H5RjCY3vl9edvGtV9YJ+rynjqfBUsBb3B5qS+WgI95IJ3dY7h4HVhKFX6s76GZ0AvfpJYngp9aZu36p1/IsW70UjwaIE7WO+HiyUGbPxoFq1xGGRbnKSALTbyzcR33LvJltGjPTuKs9eZTMUCSD0bPw3UAMfQrZ06DcdWYNZB4OYeofArGE9n4YqIkHEfV1AQgHH9SavVAvPA4LHaM= ceph-45356448-e9c2-11ee-95cd-87774f69a715 2024-03-24T09:40:40.308 DEBUG:teuthology.orchestra.run.smithi155:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDNcYlsM3foEXSl36/mFg/fy7bIaYscKfx79D+1lQ6b5DszL+AkkZZKMsTkQZjpQiKFn4gIDjgomzNLJxwJmslxl1eZpZUjHK9rHW4QACShEQdsU8Jn3rnhw9iZv6UdVEqVw/KNtO6jSdhsJaobCp8NjwNxB5sBqnuvTmSqClvHFqcaC+/cEpGrZQnjkni+kT8FlL+Mc7JrZRtRrpF/u3z2LpHUoyOcty88EyRn5vzKu4pqAuuzo9iD6of0NNhPTuI19EpjWVemSyTBi0PogQxw15sChCebC5UBJhqddUxhZ1T6H5RjCY3vl9edvGtV9YJ+rynjqfBUsBb3B5qS+WgI95IJ3dY7h4HVhKFX6s76GZ0AvfpJYngp9aZu36p1/IsW70UjwaIE7WO+HiyUGbPxoFq1xGGRbnKSALTbyzcR33LvJltGjPTuKs9eZTMUCSD0bPw3UAMfQrZ06DcdWYNZB4OYeofArGE9n4YqIkHEfV1AQgHH9SavVAvPA4LHaM= ceph-45356448-e9c2-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T09:40:40.356 INFO:teuthology.orchestra.run.smithi155.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDNcYlsM3foEXSl36/mFg/fy7bIaYscKfx79D+1lQ6b5DszL+AkkZZKMsTkQZjpQiKFn4gIDjgomzNLJxwJmslxl1eZpZUjHK9rHW4QACShEQdsU8Jn3rnhw9iZv6UdVEqVw/KNtO6jSdhsJaobCp8NjwNxB5sBqnuvTmSqClvHFqcaC+/cEpGrZQnjkni+kT8FlL+Mc7JrZRtRrpF/u3z2LpHUoyOcty88EyRn5vzKu4pqAuuzo9iD6of0NNhPTuI19EpjWVemSyTBi0PogQxw15sChCebC5UBJhqddUxhZ1T6H5RjCY3vl9edvGtV9YJ+rynjqfBUsBb3B5qS+WgI95IJ3dY7h4HVhKFX6s76GZ0AvfpJYngp9aZu36p1/IsW70UjwaIE7WO+HiyUGbPxoFq1xGGRbnKSALTbyzcR33LvJltGjPTuKs9eZTMUCSD0bPw3UAMfQrZ06DcdWYNZB4OYeofArGE9n4YqIkHEfV1AQgHH9SavVAvPA4LHaM= ceph-45356448-e9c2-11ee-95cd-87774f69a715 2024-03-24T09:40:40.371 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-24T09:40:40.571 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:40:40.693 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:40 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/1831129017' entity='client.admin' 2024-03-24T09:40:43.060 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.059+0000 7f41d7259700 1 -- 172.21.15.22:0/2424468035 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f41d0100750 msgr2=0x7f41d0100b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:43.060 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.059+0000 7f41d7259700 1 --2- 172.21.15.22:0/2424468035 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f41d0100750 0x7f41d0100b70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f41c4009910 tx=0x7f41c4009c20 comp rx=0 tx=0).stop 2024-03-24T09:40:43.060 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.059+0000 7f41d7259700 1 -- 172.21.15.22:0/2424468035 shutdown_connections 2024-03-24T09:40:43.061 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.059+0000 7f41d7259700 1 --2- 172.21.15.22:0/2424468035 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f41d0100750 0x7f41d0100b70 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:43.061 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.059+0000 7f41d7259700 1 -- 172.21.15.22:0/2424468035 >> 172.21.15.22:0/2424468035 conn(0x7f41d00fbcd0 msgr2=0x7f41d00fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:43.061 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.059+0000 7f41d7259700 1 -- 172.21.15.22:0/2424468035 shutdown_connections 2024-03-24T09:40:43.061 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.059+0000 7f41d7259700 1 -- 172.21.15.22:0/2424468035 wait complete. 2024-03-24T09:40:43.061 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.060+0000 7f41d7259700 1 Processor -- start 2024-03-24T09:40:43.061 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.060+0000 7f41d7259700 1 -- start start 2024-03-24T09:40:43.061 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.060+0000 7f41d7259700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f41d0100750 0x7f41d0195bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:43.061 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.060+0000 7f41d7259700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d0196100 con 0x7f41d0100750 2024-03-24T09:40:43.061 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.060+0000 7f41d4ff5700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f41d0100750 0x7f41d0195bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:43.061 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.060+0000 7f41d4ff5700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f41d0100750 0x7f41d0195bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60438/0 (socket says 172.21.15.22:60438) 2024-03-24T09:40:43.061 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.060+0000 7f41d4ff5700 1 -- 172.21.15.22:0/2489491006 learned_addr learned my addr 172.21.15.22:0/2489491006 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:43.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.060+0000 7f41d4ff5700 1 -- 172.21.15.22:0/2489491006 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c40095f0 con 0x7f41d0100750 2024-03-24T09:40:43.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.061+0000 7f41d4ff5700 1 --2- 172.21.15.22:0/2489491006 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f41d0100750 0x7f41d0195bc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f41c4006010 tx=0x7f41c4009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:43.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.061+0000 7f41cdffb700 1 -- 172.21.15.22:0/2489491006 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41c401d070 con 0x7f41d0100750 2024-03-24T09:40:43.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.061+0000 7f41cdffb700 1 -- 172.21.15.22:0/2489491006 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f41c4005dc0 con 0x7f41d0100750 2024-03-24T09:40:43.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.061+0000 7f41d7259700 1 -- 172.21.15.22:0/2489491006 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41d0196300 con 0x7f41d0100750 2024-03-24T09:40:43.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.061+0000 7f41cdffb700 1 -- 172.21.15.22:0/2489491006 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41c4022e90 con 0x7f41d0100750 2024-03-24T09:40:43.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.061+0000 7f41d7259700 1 -- 172.21.15.22:0/2489491006 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41d01967a0 con 0x7f41d0100750 2024-03-24T09:40:43.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.062+0000 7f41cdffb700 1 -- 172.21.15.22:0/2489491006 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 13) v1 ==== 45604+0+0 (secure 0 0 0) 0x7f41c400f4f0 con 0x7f41d0100750 2024-03-24T09:40:43.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.062+0000 7f41d7259700 1 -- 172.21.15.22:0/2489491006 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41d018f580 con 0x7f41d0100750 2024-03-24T09:40:43.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.062+0000 7f41cdffb700 1 --2- 172.21.15.22:0/2489491006 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f41c0038760 0x7f41c003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:43.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.062+0000 7f41cdffb700 1 -- 172.21.15.22:0/2489491006 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f41c404cbd0 con 0x7f41d0100750 2024-03-24T09:40:43.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.062+0000 7f41cffff700 1 --2- 172.21.15.22:0/2489491006 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f41c0038760 0x7f41c003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:43.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.063+0000 7f41cffff700 1 --2- 172.21.15.22:0/2489491006 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f41c0038760 0x7f41c003ac20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f41bc006fd0 tx=0x7f41bc006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:43.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.071+0000 7f41cdffb700 1 -- 172.21.15.22:0/2489491006 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f41c402a950 con 0x7f41d0100750 2024-03-24T09:40:43.180 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:42 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:43.180 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:42 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:43.180 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:42 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config rm", "who": "osd/host:smithi022", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:40:43.180 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:42 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:43.180 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:42 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi022", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-24T09:40:43.180 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:42 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi022", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-03-24T09:40:43.180 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:42 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:40:43.180 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:42 smithi022 ceph-mon[29512]: Deploying daemon ceph-exporter.smithi022 on smithi022 2024-03-24T09:40:43.282 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.282+0000 7f41d7259700 1 -- 172.21.15.22:0/2489491006 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f41d00623e0 con 0x7f41d0100750 2024-03-24T09:40:43.285 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.285+0000 7f41cdffb700 1 -- 172.21.15.22:0/2489491006 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f41d00623e0 con 0x7f41d0100750 2024-03-24T09:40:43.288 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.287+0000 7f41d7259700 1 -- 172.21.15.22:0/2489491006 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f41c0038760 msgr2=0x7f41c003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:43.288 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.287+0000 7f41d7259700 1 --2- 172.21.15.22:0/2489491006 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f41c0038760 0x7f41c003ac20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f41bc006fd0 tx=0x7f41bc006e40 comp rx=0 tx=0).stop 2024-03-24T09:40:43.288 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.287+0000 7f41d7259700 1 -- 172.21.15.22:0/2489491006 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f41d0100750 msgr2=0x7f41d0195bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:43.288 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.287+0000 7f41d7259700 1 --2- 172.21.15.22:0/2489491006 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f41d0100750 0x7f41d0195bc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f41c4006010 tx=0x7f41c4009f80 comp rx=0 tx=0).stop 2024-03-24T09:40:43.288 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.288+0000 7f41d7259700 1 -- 172.21.15.22:0/2489491006 shutdown_connections 2024-03-24T09:40:43.288 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.288+0000 7f41d7259700 1 --2- 172.21.15.22:0/2489491006 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f41c0038760 0x7f41c003ac20 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:43.288 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.288+0000 7f41d7259700 1 --2- 172.21.15.22:0/2489491006 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f41d0100750 0x7f41d0195bc0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:43.288 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.288+0000 7f41d7259700 1 -- 172.21.15.22:0/2489491006 >> 172.21.15.22:0/2489491006 conn(0x7f41d00fbcd0 msgr2=0x7f41d00fc740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:43.288 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.288+0000 7f41d7259700 1 -- 172.21.15.22:0/2489491006 shutdown_connections 2024-03-24T09:40:43.289 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:43.288+0000 7f41d7259700 1 -- 172.21.15.22:0/2489491006 wait complete. 2024-03-24T09:40:43.880 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-24T09:40:43.881 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-24T09:40:44.137 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:40:44.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:44 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/2489491006' entity='client.admin' 2024-03-24T09:40:46.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.182+0000 7f4672a29700 1 -- 172.21.15.22:0/1696783837 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f466c0ff640 msgr2=0x7f466c0ffa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:46.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.182+0000 7f4672a29700 1 --2- 172.21.15.22:0/1696783837 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f466c0ff640 0x7f466c0ffa60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f465c009910 tx=0x7f465c009c20 comp rx=0 tx=0).stop 2024-03-24T09:40:46.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.182+0000 7f4672a29700 1 -- 172.21.15.22:0/1696783837 shutdown_connections 2024-03-24T09:40:46.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.182+0000 7f4672a29700 1 --2- 172.21.15.22:0/1696783837 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f466c0ff640 0x7f466c0ffa60 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:46.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.182+0000 7f4672a29700 1 -- 172.21.15.22:0/1696783837 >> 172.21.15.22:0/1696783837 conn(0x7f466c0fabe0 msgr2=0x7f466c0fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:46.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.182+0000 7f4672a29700 1 -- 172.21.15.22:0/1696783837 shutdown_connections 2024-03-24T09:40:46.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.182+0000 7f4672a29700 1 -- 172.21.15.22:0/1696783837 wait complete. 2024-03-24T09:40:46.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.182+0000 7f4672a29700 1 Processor -- start 2024-03-24T09:40:46.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.183+0000 7f4672a29700 1 -- start start 2024-03-24T09:40:46.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.183+0000 7f4672a29700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f466c0ff640 0x7f466c194ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:46.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.183+0000 7f4672a29700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466c195020 con 0x7f466c0ff640 2024-03-24T09:40:46.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.183+0000 7f466bfff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f466c0ff640 0x7f466c194ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:46.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.183+0000 7f466bfff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f466c0ff640 0x7f466c194ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60450/0 (socket says 172.21.15.22:60450) 2024-03-24T09:40:46.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.183+0000 7f466bfff700 1 -- 172.21.15.22:0/2734251839 learned_addr learned my addr 172.21.15.22:0/2734251839 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:46.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.183+0000 7f466bfff700 1 -- 172.21.15.22:0/2734251839 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f465c0095f0 con 0x7f466c0ff640 2024-03-24T09:40:46.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.183+0000 7f466bfff700 1 --2- 172.21.15.22:0/2734251839 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f466c0ff640 0x7f466c194ae0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f465c006010 tx=0x7f465c009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:46.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.183+0000 7f46697fa700 1 -- 172.21.15.22:0/2734251839 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f465c01d070 con 0x7f466c0ff640 2024-03-24T09:40:46.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.184+0000 7f46697fa700 1 -- 172.21.15.22:0/2734251839 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f465c005dc0 con 0x7f466c0ff640 2024-03-24T09:40:46.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.184+0000 7f4672a29700 1 -- 172.21.15.22:0/2734251839 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f466c195220 con 0x7f466c0ff640 2024-03-24T09:40:46.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.184+0000 7f46697fa700 1 -- 172.21.15.22:0/2734251839 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f465c022e90 con 0x7f466c0ff640 2024-03-24T09:40:46.186 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.184+0000 7f4672a29700 1 -- 172.21.15.22:0/2734251839 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f466c1956c0 con 0x7f466c0ff640 2024-03-24T09:40:46.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.184+0000 7f46697fa700 1 -- 172.21.15.22:0/2734251839 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 13) v1 ==== 45604+0+0 (secure 0 0 0) 0x7f465c022470 con 0x7f466c0ff640 2024-03-24T09:40:46.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.185+0000 7f4672a29700 1 -- 172.21.15.22:0/2734251839 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f466c18e470 con 0x7f466c0ff640 2024-03-24T09:40:46.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.185+0000 7f46697fa700 1 --2- 172.21.15.22:0/2734251839 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f46540387c0 0x7f465403ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:46.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.185+0000 7f46697fa700 1 -- 172.21.15.22:0/2734251839 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f465c04c790 con 0x7f466c0ff640 2024-03-24T09:40:46.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.185+0000 7f466b7fe700 1 --2- 172.21.15.22:0/2734251839 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f46540387c0 0x7f465403ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:46.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.185+0000 7f466b7fe700 1 --2- 172.21.15.22:0/2734251839 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f46540387c0 0x7f465403ac80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4660006fd0 tx=0x7f4660006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:46.190 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.190+0000 7f46697fa700 1 -- 172.21.15.22:0/2734251839 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f465c00b960 con 0x7f466c0ff640 2024-03-24T09:40:46.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.396+0000 7f4672a29700 1 -- 172.21.15.22:0/2734251839 --> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f466c0611f0 con 0x7f46540387c0 2024-03-24T09:40:46.400 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.399+0000 7f46697fa700 1 -- 172.21.15.22:0/2734251839 <== mgr.14164 v2:172.21.15.22:6800/2979399337 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f466c0611f0 con 0x7f46540387c0 2024-03-24T09:40:46.402 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.402+0000 7f4672a29700 1 -- 172.21.15.22:0/2734251839 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f46540387c0 msgr2=0x7f465403ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:46.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.402+0000 7f4672a29700 1 --2- 172.21.15.22:0/2734251839 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f46540387c0 0x7f465403ac80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4660006fd0 tx=0x7f4660006e40 comp rx=0 tx=0).stop 2024-03-24T09:40:46.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.402+0000 7f4672a29700 1 -- 172.21.15.22:0/2734251839 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f466c0ff640 msgr2=0x7f466c194ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:46.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.402+0000 7f4672a29700 1 --2- 172.21.15.22:0/2734251839 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f466c0ff640 0x7f466c194ae0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f465c006010 tx=0x7f465c009f80 comp rx=0 tx=0).stop 2024-03-24T09:40:46.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.402+0000 7f4672a29700 1 -- 172.21.15.22:0/2734251839 shutdown_connections 2024-03-24T09:40:46.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.402+0000 7f4672a29700 1 --2- 172.21.15.22:0/2734251839 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f46540387c0 0x7f465403ac80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:46.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.402+0000 7f4672a29700 1 --2- 172.21.15.22:0/2734251839 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f466c0ff640 0x7f466c194ae0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:46.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.402+0000 7f4672a29700 1 -- 172.21.15.22:0/2734251839 >> 172.21.15.22:0/2734251839 conn(0x7f466c0fabe0 msgr2=0x7f466c0fb670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:46.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.402+0000 7f4672a29700 1 -- 172.21.15.22:0/2734251839 shutdown_connections 2024-03-24T09:40:46.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:46.402+0000 7f4672a29700 1 -- 172.21.15.22:0/2734251839 wait complete. 2024-03-24T09:40:47.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:47 smithi022 ceph-mon[29512]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:40:47.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:47 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:47.568 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi155 2024-03-24T09:40:47.568 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-24T09:40:47.568 DEBUG:teuthology.orchestra.run.smithi155:> dd of=/etc/ceph/ceph.conf 2024-03-24T09:40:47.587 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-24T09:40:47.587 DEBUG:teuthology.orchestra.run.smithi155:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T09:40:47.646 INFO:tasks.cephadm:Adding host smithi155 to orchestrator... 2024-03-24T09:40:47.646 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph orch host add smithi155 2024-03-24T09:40:48.082 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:40:49.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:49 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:49.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:49 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:49.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:49 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:49.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:49 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:49.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:49 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi022", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T09:40:49.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:49 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi022", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-24T09:40:49.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:49 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:40:49.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:49 smithi022 ceph-mon[29512]: Deploying daemon crash.smithi022 on smithi022 2024-03-24T09:40:50.240 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:50 smithi022 ceph-mon[29512]: from='client.14191 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi155", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:40:50.437 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:49.686+0000 7f6195db1700 1 -- 172.21.15.22:0/103562906 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f61900ff640 msgr2=0x7f61900ffa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:50.437 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:49.686+0000 7f6195db1700 1 --2- 172.21.15.22:0/103562906 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f61900ff640 0x7f61900ffa60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6180009910 tx=0x7f6180009c20 comp rx=0 tx=0).stop 2024-03-24T09:40:50.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:49.686+0000 7f6195db1700 1 -- 172.21.15.22:0/103562906 shutdown_connections 2024-03-24T09:40:50.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:49.686+0000 7f6195db1700 1 --2- 172.21.15.22:0/103562906 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f61900ff640 0x7f61900ffa60 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:50.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:49.686+0000 7f6195db1700 1 -- 172.21.15.22:0/103562906 >> 172.21.15.22:0/103562906 conn(0x7f61900fabe0 msgr2=0x7f61900fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:50.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:49.686+0000 7f6195db1700 1 -- 172.21.15.22:0/103562906 shutdown_connections 2024-03-24T09:40:50.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:49.686+0000 7f6195db1700 1 -- 172.21.15.22:0/103562906 wait complete. 2024-03-24T09:40:50.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:49.686+0000 7f6195db1700 1 Processor -- start 2024-03-24T09:40:50.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:49.687+0000 7f6195db1700 1 -- start start 2024-03-24T09:40:50.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:49.687+0000 7f6195db1700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f61900ff640 0x7f6190195bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:50.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:49.687+0000 7f6195db1700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6190196100 con 0x7f61900ff640 2024-03-24T09:40:50.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:49.687+0000 7f618f7fe700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f61900ff640 0x7f6190195bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:50.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:49.687+0000 7f618f7fe700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f61900ff640 0x7f6190195bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:43904/0 (socket says 172.21.15.22:43904) 2024-03-24T09:40:50.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:49.687+0000 7f618f7fe700 1 -- 172.21.15.22:0/954284399 learned_addr learned my addr 172.21.15.22:0/954284399 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:50.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:49.687+0000 7f618f7fe700 1 -- 172.21.15.22:0/954284399 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61800095f0 con 0x7f61900ff640 2024-03-24T09:40:50.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:49.688+0000 7f618f7fe700 1 --2- 172.21.15.22:0/954284399 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f61900ff640 0x7f6190195bc0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6180006010 tx=0x7f6180009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:50.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:49.688+0000 7f618cff9700 1 -- 172.21.15.22:0/954284399 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f618001d070 con 0x7f61900ff640 2024-03-24T09:40:50.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:49.688+0000 7f618cff9700 1 -- 172.21.15.22:0/954284399 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6180005dc0 con 0x7f61900ff640 2024-03-24T09:40:50.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:49.688+0000 7f6195db1700 1 -- 172.21.15.22:0/954284399 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6190196300 con 0x7f61900ff640 2024-03-24T09:40:50.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:49.688+0000 7f618cff9700 1 -- 172.21.15.22:0/954284399 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6180022e90 con 0x7f61900ff640 2024-03-24T09:40:50.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:49.688+0000 7f6195db1700 1 -- 172.21.15.22:0/954284399 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61901967a0 con 0x7f61900ff640 2024-03-24T09:40:50.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:49.689+0000 7f618cff9700 1 -- 172.21.15.22:0/954284399 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 13) v1 ==== 45604+0+0 (secure 0 0 0) 0x7f618000f4f0 con 0x7f61900ff640 2024-03-24T09:40:50.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:49.689+0000 7f6195db1700 1 -- 172.21.15.22:0/954284399 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f619018f6f0 con 0x7f61900ff640 2024-03-24T09:40:50.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:49.689+0000 7f618cff9700 1 --2- 172.21.15.22:0/954284399 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f6178038760 0x7f617803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:50.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:49.689+0000 7f618cff9700 1 -- 172.21.15.22:0/954284399 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f618004c810 con 0x7f61900ff640 2024-03-24T09:40:50.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:49.689+0000 7f618effd700 1 --2- 172.21.15.22:0/954284399 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f6178038760 0x7f617803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:50.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:49.690+0000 7f618effd700 1 --2- 172.21.15.22:0/954284399 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f6178038760 0x7f617803ac20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6184006fd0 tx=0x7f6184006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:50.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:49.694+0000 7f618cff9700 1 -- 172.21.15.22:0/954284399 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f618002a950 con 0x7f61900ff640 2024-03-24T09:40:50.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:49.902+0000 7f6195db1700 1 -- 172.21.15.22:0/954284399 --> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi155", "target": ["mon-mgr", ""]}) v1 -- 0x7f61900611f0 con 0x7f6178038760 2024-03-24T09:40:52.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:52 smithi022 ceph-mon[29512]: Deploying cephadm binary to smithi155 2024-03-24T09:40:52.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:52 smithi022 ceph-mon[29512]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:40:52.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:52 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:52.619 INFO:teuthology.orchestra.run.smithi022.stdout:Added host 'smithi155' with addr '172.21.15.155' 2024-03-24T09:40:52.620 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:52.618+0000 7f618cff9700 1 -- 172.21.15.22:0/954284399 <== mgr.14164 v2:172.21.15.22:6800/2979399337 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f61900611f0 con 0x7f6178038760 2024-03-24T09:40:52.621 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:52.621+0000 7f6195db1700 1 -- 172.21.15.22:0/954284399 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f6178038760 msgr2=0x7f617803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:52.621 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:52.621+0000 7f6195db1700 1 --2- 172.21.15.22:0/954284399 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f6178038760 0x7f617803ac20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6184006fd0 tx=0x7f6184006e40 comp rx=0 tx=0).stop 2024-03-24T09:40:52.621 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:52.621+0000 7f6195db1700 1 -- 172.21.15.22:0/954284399 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f61900ff640 msgr2=0x7f6190195bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:52.621 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:52.621+0000 7f6195db1700 1 --2- 172.21.15.22:0/954284399 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f61900ff640 0x7f6190195bc0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6180006010 tx=0x7f6180009f80 comp rx=0 tx=0).stop 2024-03-24T09:40:52.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:52.621+0000 7f6195db1700 1 -- 172.21.15.22:0/954284399 shutdown_connections 2024-03-24T09:40:52.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:52.621+0000 7f6195db1700 1 --2- 172.21.15.22:0/954284399 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f6178038760 0x7f617803ac20 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:52.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:52.621+0000 7f6195db1700 1 --2- 172.21.15.22:0/954284399 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f61900ff640 0x7f6190195bc0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:52.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:52.621+0000 7f6195db1700 1 -- 172.21.15.22:0/954284399 >> 172.21.15.22:0/954284399 conn(0x7f61900fabe0 msgr2=0x7f6190068c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:52.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:52.621+0000 7f6195db1700 1 -- 172.21.15.22:0/954284399 shutdown_connections 2024-03-24T09:40:52.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:52.621+0000 7f6195db1700 1 -- 172.21.15.22:0/954284399 wait complete. 2024-03-24T09:40:53.255 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:53 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:53.255 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:53 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:53.255 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:53 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:53.255 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:53 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:53.255 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:53 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:53.279 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-24T09:40:53.493 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:40:54.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:54 smithi022 ceph-mon[29512]: Deploying daemon node-exporter.smithi022 on smithi022 2024-03-24T09:40:54.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:54 smithi022 ceph-mon[29512]: Added host smithi155 2024-03-24T09:40:54.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:54 smithi022 ceph-mon[29512]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:40:56.012 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:55.940+0000 7f87a306f700 1 -- 172.21.15.22:0/2720496564 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f879c1011c0 msgr2=0x7f879c1035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:56.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:55.940+0000 7f87a306f700 1 --2- 172.21.15.22:0/2720496564 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f879c1011c0 0x7f879c1035b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8790009910 tx=0x7f8790009c20 comp rx=0 tx=0).stop 2024-03-24T09:40:56.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:55.940+0000 7f87a306f700 1 -- 172.21.15.22:0/2720496564 shutdown_connections 2024-03-24T09:40:56.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:55.940+0000 7f87a306f700 1 --2- 172.21.15.22:0/2720496564 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f879c1011c0 0x7f879c1035b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:56.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:55.940+0000 7f87a306f700 1 -- 172.21.15.22:0/2720496564 >> 172.21.15.22:0/2720496564 conn(0x7f879c0fabd0 msgr2=0x7f879c0fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:56.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:55.940+0000 7f87a306f700 1 -- 172.21.15.22:0/2720496564 shutdown_connections 2024-03-24T09:40:56.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:55.940+0000 7f87a306f700 1 -- 172.21.15.22:0/2720496564 wait complete. 2024-03-24T09:40:56.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:55.940+0000 7f87a306f700 1 Processor -- start 2024-03-24T09:40:56.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:55.940+0000 7f87a306f700 1 -- start start 2024-03-24T09:40:56.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:55.940+0000 7f87a306f700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f879c1011c0 0x7f879c195c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:56.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:55.940+0000 7f87a306f700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f879c1961b0 con 0x7f879c1011c0 2024-03-24T09:40:56.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:55.940+0000 7f87a0e0b700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f879c1011c0 0x7f879c195c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:56.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:55.941+0000 7f87a0e0b700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f879c1011c0 0x7f879c195c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:43940/0 (socket says 172.21.15.22:43940) 2024-03-24T09:40:56.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:55.941+0000 7f87a0e0b700 1 -- 172.21.15.22:0/1559167928 learned_addr learned my addr 172.21.15.22:0/1559167928 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:56.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:55.941+0000 7f87a0e0b700 1 -- 172.21.15.22:0/1559167928 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87900095f0 con 0x7f879c1011c0 2024-03-24T09:40:56.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:55.941+0000 7f87a0e0b700 1 --2- 172.21.15.22:0/1559167928 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f879c1011c0 0x7f879c195c70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8790005b40 tx=0x7f8790005000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:56.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:55.941+0000 7f8799ffb700 1 -- 172.21.15.22:0/1559167928 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f879001d070 con 0x7f879c1011c0 2024-03-24T09:40:56.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:55.941+0000 7f8799ffb700 1 -- 172.21.15.22:0/1559167928 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8790005dc0 con 0x7f879c1011c0 2024-03-24T09:40:56.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:55.941+0000 7f87a306f700 1 -- 172.21.15.22:0/1559167928 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f879c1963b0 con 0x7f879c1011c0 2024-03-24T09:40:56.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:55.941+0000 7f8799ffb700 1 -- 172.21.15.22:0/1559167928 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8790022e00 con 0x7f879c1011c0 2024-03-24T09:40:56.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:55.941+0000 7f87a306f700 1 -- 172.21.15.22:0/1559167928 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f879c196850 con 0x7f879c1011c0 2024-03-24T09:40:56.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:55.942+0000 7f8799ffb700 1 -- 172.21.15.22:0/1559167928 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 13) v1 ==== 45604+0+0 (secure 0 0 0) 0x7f879000f4b0 con 0x7f879c1011c0 2024-03-24T09:40:56.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:55.942+0000 7f8799ffb700 1 --2- 172.21.15.22:0/1559167928 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f878c038760 0x7f878c03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:56.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:55.942+0000 7f8799ffb700 1 -- 172.21.15.22:0/1559167928 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f879004c890 con 0x7f879c1011c0 2024-03-24T09:40:56.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:55.942+0000 7f87a306f700 1 -- 172.21.15.22:0/1559167928 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f879c18f560 con 0x7f879c1011c0 2024-03-24T09:40:56.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:55.942+0000 7f879bfff700 1 --2- 172.21.15.22:0/1559167928 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f878c038760 0x7f878c03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:56.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:55.943+0000 7f879bfff700 1 --2- 172.21.15.22:0/1559167928 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f878c038760 0x7f878c03ac20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8788006fd0 tx=0x7f8788006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:56.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:55.947+0000 7f8799ffb700 1 -- 172.21.15.22:0/1559167928 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8790027020 con 0x7f879c1011c0 2024-03-24T09:40:56.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:56.157+0000 7f87a306f700 1 -- 172.21.15.22:0/1559167928 --> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f879c0611f0 con 0x7f878c038760 2024-03-24T09:40:56.157 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:56 smithi022 ceph-mon[29512]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:40:56.157 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:56 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:56.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:56.158+0000 7f8799ffb700 1 -- 172.21.15.22:0/1559167928 <== mgr.14164 v2:172.21.15.22:6800/2979399337 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f879c0611f0 con 0x7f878c038760 2024-03-24T09:40:56.158 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:40:56.158 INFO:teuthology.orchestra.run.smithi022.stdout:[{"addr": "172.21.15.22", "hostname": "smithi022", "labels": [], "status": ""}, {"addr": "172.21.15.155", "hostname": "smithi155", "labels": [], "status": ""}] 2024-03-24T09:40:56.160 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:56.160+0000 7f87a306f700 1 -- 172.21.15.22:0/1559167928 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f878c038760 msgr2=0x7f878c03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:56.161 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:56.160+0000 7f87a306f700 1 --2- 172.21.15.22:0/1559167928 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f878c038760 0x7f878c03ac20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8788006fd0 tx=0x7f8788006e40 comp rx=0 tx=0).stop 2024-03-24T09:40:56.161 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:56.160+0000 7f87a306f700 1 -- 172.21.15.22:0/1559167928 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f879c1011c0 msgr2=0x7f879c195c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:56.161 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:56.160+0000 7f87a306f700 1 --2- 172.21.15.22:0/1559167928 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f879c1011c0 0x7f879c195c70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8790005b40 tx=0x7f8790005000 comp rx=0 tx=0).stop 2024-03-24T09:40:56.161 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:56.160+0000 7f87a306f700 1 -- 172.21.15.22:0/1559167928 shutdown_connections 2024-03-24T09:40:56.161 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:56.160+0000 7f87a306f700 1 --2- 172.21.15.22:0/1559167928 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f878c038760 0x7f878c03ac20 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:56.161 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:56.160+0000 7f87a306f700 1 --2- 172.21.15.22:0/1559167928 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f879c1011c0 0x7f879c195c70 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:56.161 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:56.160+0000 7f87a306f700 1 -- 172.21.15.22:0/1559167928 >> 172.21.15.22:0/1559167928 conn(0x7f879c0fabd0 msgr2=0x7f879c0fb740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:56.161 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:56.160+0000 7f87a306f700 1 -- 172.21.15.22:0/1559167928 shutdown_connections 2024-03-24T09:40:56.161 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:56.160+0000 7f87a306f700 1 -- 172.21.15.22:0/1559167928 wait complete. 2024-03-24T09:40:56.996 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-24T09:40:56.996 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph osd crush tunables default 2024-03-24T09:40:57.270 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:40:58.270 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:58 smithi022 ceph-mon[29512]: from='client.14193 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T09:40:58.271 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:58 smithi022 ceph-mon[29512]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:40:58.271 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:58 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:58.271 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:58 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:58.271 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:58 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:58.271 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:58 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:40:58.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:58.726+0000 7fc7afa61700 1 -- 172.21.15.22:0/2244924488 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc7a80fea10 msgr2=0x7fc7a80fee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:58.728 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:58.726+0000 7fc7afa61700 1 --2- 172.21.15.22:0/2244924488 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc7a80fea10 0x7fc7a80fee30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc7a4009910 tx=0x7fc7a4009c20 comp rx=0 tx=0).stop 2024-03-24T09:40:58.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:58.726+0000 7fc7afa61700 1 -- 172.21.15.22:0/2244924488 shutdown_connections 2024-03-24T09:40:58.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:58.726+0000 7fc7afa61700 1 --2- 172.21.15.22:0/2244924488 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc7a80fea10 0x7fc7a80fee30 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:58.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:58.726+0000 7fc7afa61700 1 -- 172.21.15.22:0/2244924488 >> 172.21.15.22:0/2244924488 conn(0x7fc7a80fa350 msgr2=0x7fc7a80fc7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:58.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:58.726+0000 7fc7afa61700 1 -- 172.21.15.22:0/2244924488 shutdown_connections 2024-03-24T09:40:58.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:58.726+0000 7fc7afa61700 1 -- 172.21.15.22:0/2244924488 wait complete. 2024-03-24T09:40:58.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:58.727+0000 7fc7afa61700 1 Processor -- start 2024-03-24T09:40:58.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:58.727+0000 7fc7afa61700 1 -- start start 2024-03-24T09:40:58.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:58.727+0000 7fc7afa61700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc7a80fea10 0x7fc7a8101220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:58.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:58.727+0000 7fc7afa61700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a8101760 con 0x7fc7a80fea10 2024-03-24T09:40:58.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:58.727+0000 7fc7ad7fd700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc7a80fea10 0x7fc7a8101220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:58.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:58.727+0000 7fc7ad7fd700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc7a80fea10 0x7fc7a8101220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:48258/0 (socket says 172.21.15.22:48258) 2024-03-24T09:40:58.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:58.727+0000 7fc7ad7fd700 1 -- 172.21.15.22:0/3227038251 learned_addr learned my addr 172.21.15.22:0/3227038251 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:40:58.729 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:58.727+0000 7fc7ad7fd700 1 -- 172.21.15.22:0/3227038251 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a40095f0 con 0x7fc7a80fea10 2024-03-24T09:40:58.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:58.727+0000 7fc7ad7fd700 1 --2- 172.21.15.22:0/3227038251 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc7a80fea10 0x7fc7a8101220 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc7a4006010 tx=0x7fc7a40050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:58.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:58.728+0000 7fc79e7fc700 1 -- 172.21.15.22:0/3227038251 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7a401d070 con 0x7fc7a80fea10 2024-03-24T09:40:58.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:58.728+0000 7fc7afa61700 1 -- 172.21.15.22:0/3227038251 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7a80ff9a0 con 0x7fc7a80fea10 2024-03-24T09:40:58.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:58.728+0000 7fc79e7fc700 1 -- 172.21.15.22:0/3227038251 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc7a400be20 con 0x7fc7a80fea10 2024-03-24T09:40:58.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:58.728+0000 7fc79e7fc700 1 -- 172.21.15.22:0/3227038251 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7a4022de0 con 0x7fc7a80fea10 2024-03-24T09:40:58.730 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:58.728+0000 7fc7afa61700 1 -- 172.21.15.22:0/3227038251 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7a80ffe40 con 0x7fc7a80fea10 2024-03-24T09:40:58.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:58.728+0000 7fc79e7fc700 1 -- 172.21.15.22:0/3227038251 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 13) v1 ==== 45604+0+0 (secure 0 0 0) 0x7fc7a40226c0 con 0x7fc7a80fea10 2024-03-24T09:40:58.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:58.729+0000 7fc7afa61700 1 -- 172.21.15.22:0/3227038251 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7a8106ec0 con 0x7fc7a80fea10 2024-03-24T09:40:58.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:58.729+0000 7fc79e7fc700 1 --2- 172.21.15.22:0/3227038251 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7fc7940387b0 0x7fc79403ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:40:58.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:58.729+0000 7fc79e7fc700 1 -- 172.21.15.22:0/3227038251 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc7a404c7f0 con 0x7fc7a80fea10 2024-03-24T09:40:58.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:58.729+0000 7fc7acffc700 1 --2- 172.21.15.22:0/3227038251 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7fc7940387b0 0x7fc79403ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:40:58.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:58.729+0000 7fc7acffc700 1 --2- 172.21.15.22:0/3227038251 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7fc7940387b0 0x7fc79403ac70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc798006fd0 tx=0x7fc798006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:40:58.734 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:58.734+0000 7fc79e7fc700 1 -- 172.21.15.22:0/3227038251 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc7a400baa0 con 0x7fc7a80fea10 2024-03-24T09:40:58.948 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:58.947+0000 7fc7afa61700 1 -- 172.21.15.22:0/3227038251 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fc7a80ff540 con 0x7fc7a80fea10 2024-03-24T09:40:59.011 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:59.011+0000 7fc79e7fc700 1 -- 172.21.15.22:0/3227038251 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fc7a80ff540 con 0x7fc7a80fea10 2024-03-24T09:40:59.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:59.013+0000 7fc7afa61700 1 -- 172.21.15.22:0/3227038251 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7fc7940387b0 msgr2=0x7fc79403ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:59.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:59.013+0000 7fc7afa61700 1 --2- 172.21.15.22:0/3227038251 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7fc7940387b0 0x7fc79403ac70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc798006fd0 tx=0x7fc798006e40 comp rx=0 tx=0).stop 2024-03-24T09:40:59.013 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:59.013+0000 7fc7afa61700 1 -- 172.21.15.22:0/3227038251 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc7a80fea10 msgr2=0x7fc7a8101220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:40:59.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:59.013+0000 7fc7afa61700 1 --2- 172.21.15.22:0/3227038251 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc7a80fea10 0x7fc7a8101220 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc7a4006010 tx=0x7fc7a40050b0 comp rx=0 tx=0).stop 2024-03-24T09:40:59.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:59.013+0000 7fc7afa61700 1 -- 172.21.15.22:0/3227038251 shutdown_connections 2024-03-24T09:40:59.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:59.013+0000 7fc7afa61700 1 --2- 172.21.15.22:0/3227038251 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7fc7940387b0 0x7fc79403ac70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:59.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:59.013+0000 7fc7afa61700 1 --2- 172.21.15.22:0/3227038251 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc7a80fea10 0x7fc7a8101220 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:40:59.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:59.013+0000 7fc7afa61700 1 -- 172.21.15.22:0/3227038251 >> 172.21.15.22:0/3227038251 conn(0x7fc7a80fa350 msgr2=0x7fc7a80fae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:40:59.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:59.013+0000 7fc7afa61700 1 -- 172.21.15.22:0/3227038251 shutdown_connections 2024-03-24T09:40:59.014 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:40:59.013+0000 7fc7afa61700 1 -- 172.21.15.22:0/3227038251 wait complete. 2024-03-24T09:40:59.014 INFO:teuthology.orchestra.run.smithi022.stderr:adjusted tunables profile to default 2024-03-24T09:40:59.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:59 smithi022 ceph-mon[29512]: Deploying daemon alertmanager.smithi022 on smithi022 2024-03-24T09:40:59.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:40:59 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/3227038251' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T09:40:59.895 INFO:tasks.cephadm:Adding mon.smithi022 on smithi022 2024-03-24T09:40:59.895 INFO:tasks.cephadm:Adding mon.smithi155 on smithi155 2024-03-24T09:40:59.896 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph orch apply mon '2;smithi022:172.21.15.22=smithi022;smithi155:172.21.15.155=smithi155' 2024-03-24T09:41:00.095 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T09:41:00.141 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T09:41:00.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:00 smithi022 ceph-mon[29512]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:41:00.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:00 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/3227038251' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T09:41:00.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:00 smithi022 ceph-mon[29512]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T09:41:01.623 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.619+0000 7f0df342d700 1 -- 172.21.15.155:0/4102086152 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0dec0ff640 msgr2=0x7f0dec0ffa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:01.624 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.619+0000 7f0df342d700 1 --2- 172.21.15.155:0/4102086152 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0dec0ff640 0x7f0dec0ffa60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0de800b0a0 tx=0x7f0de800b3b0 comp rx=0 tx=0).stop 2024-03-24T09:41:01.624 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.619+0000 7f0df342d700 1 -- 172.21.15.155:0/4102086152 shutdown_connections 2024-03-24T09:41:01.624 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.619+0000 7f0df342d700 1 --2- 172.21.15.155:0/4102086152 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0dec0ff640 0x7f0dec0ffa60 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:01.624 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.619+0000 7f0df342d700 1 -- 172.21.15.155:0/4102086152 >> 172.21.15.155:0/4102086152 conn(0x7f0dec0fabe0 msgr2=0x7f0dec0fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:41:01.624 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.620+0000 7f0df342d700 1 -- 172.21.15.155:0/4102086152 shutdown_connections 2024-03-24T09:41:01.624 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.620+0000 7f0df342d700 1 -- 172.21.15.155:0/4102086152 wait complete. 2024-03-24T09:41:01.624 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.620+0000 7f0df342d700 1 Processor -- start 2024-03-24T09:41:01.624 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.620+0000 7f0df342d700 1 -- start start 2024-03-24T09:41:01.624 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.620+0000 7f0df342d700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0dec0ff640 0x7f0dec194ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:41:01.624 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.620+0000 7f0df342d700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dec195020 con 0x7f0dec0ff640 2024-03-24T09:41:01.624 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.620+0000 7f0df11c9700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0dec0ff640 0x7f0dec194ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:41:01.624 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.620+0000 7f0df11c9700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0dec0ff640 0x7f0dec194ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.155:56700/0 (socket says 172.21.15.155:56700) 2024-03-24T09:41:01.625 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.620+0000 7f0df11c9700 1 -- 172.21.15.155:0/1092857587 learned_addr learned my addr 172.21.15.155:0/1092857587 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-24T09:41:01.625 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.621+0000 7f0df11c9700 1 -- 172.21.15.155:0/1092857587 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0de8009d00 con 0x7f0dec0ff640 2024-03-24T09:41:01.625 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.621+0000 7f0df11c9700 1 --2- 172.21.15.155:0/1092857587 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0dec0ff640 0x7f0dec194ae0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0de800bc90 tx=0x7f0de800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:41:01.625 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.621+0000 7f0de27fc700 1 -- 172.21.15.155:0/1092857587 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0de800e040 con 0x7f0dec0ff640 2024-03-24T09:41:01.625 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.621+0000 7f0df342d700 1 -- 172.21.15.155:0/1092857587 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dec195220 con 0x7f0dec0ff640 2024-03-24T09:41:01.625 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.621+0000 7f0de27fc700 1 -- 172.21.15.155:0/1092857587 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0de8003d50 con 0x7f0dec0ff640 2024-03-24T09:41:01.625 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.621+0000 7f0de27fc700 1 -- 172.21.15.155:0/1092857587 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0de801c8f0 con 0x7f0dec0ff640 2024-03-24T09:41:01.625 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.621+0000 7f0df342d700 1 -- 172.21.15.155:0/1092857587 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dec1956c0 con 0x7f0dec0ff640 2024-03-24T09:41:01.625 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.622+0000 7f0de27fc700 1 -- 172.21.15.155:0/1092857587 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 13) v1 ==== 45604+0+0 (secure 0 0 0) 0x7f0de801cbb0 con 0x7f0dec0ff640 2024-03-24T09:41:01.625 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.622+0000 7f0df342d700 1 -- 172.21.15.155:0/1092857587 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dec18e470 con 0x7f0dec0ff640 2024-03-24T09:41:01.625 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.622+0000 7f0de27fc700 1 --2- 172.21.15.155:0/1092857587 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f0dd8038810 0x7f0dd803acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:41:01.625 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.622+0000 7f0de27fc700 1 -- 172.21.15.155:0/1092857587 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0de804cab0 con 0x7f0dec0ff640 2024-03-24T09:41:01.625 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.622+0000 7f0df09c8700 1 --2- 172.21.15.155:0/1092857587 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f0dd8038810 0x7f0dd803acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:41:01.625 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.623+0000 7f0df09c8700 1 --2- 172.21.15.155:0/1092857587 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f0dd8038810 0x7f0dd803acd0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc006fd0 tx=0x7f0ddc006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:41:01.628 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.627+0000 7f0de27fc700 1 -- 172.21.15.155:0/1092857587 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0de8024360 con 0x7f0dec0ff640 2024-03-24T09:41:01.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.829+0000 7f0df342d700 1 -- 172.21.15.155:0/1092857587 --> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi022:172.21.15.22=smithi022;smithi155:172.21.15.155=smithi155", "target": ["mon-mgr", ""]}) v1 -- 0x7f0dec0612a0 con 0x7f0dd8038810 2024-03-24T09:41:01.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.834+0000 7f0de27fc700 1 -- 172.21.15.155:0/1092857587 <== mgr.14164 v2:172.21.15.22:6800/2979399337 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f0dec0612a0 con 0x7f0dd8038810 2024-03-24T09:41:01.835 INFO:teuthology.orchestra.run.smithi155.stdout:Scheduled mon update... 2024-03-24T09:41:01.837 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.836+0000 7f0df342d700 1 -- 172.21.15.155:0/1092857587 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f0dd8038810 msgr2=0x7f0dd803acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:01.837 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.836+0000 7f0df342d700 1 --2- 172.21.15.155:0/1092857587 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f0dd8038810 0x7f0dd803acd0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc006fd0 tx=0x7f0ddc006e40 comp rx=0 tx=0).stop 2024-03-24T09:41:01.837 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.836+0000 7f0df342d700 1 -- 172.21.15.155:0/1092857587 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0dec0ff640 msgr2=0x7f0dec194ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:01.837 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.836+0000 7f0df342d700 1 --2- 172.21.15.155:0/1092857587 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0dec0ff640 0x7f0dec194ae0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0de800bc90 tx=0x7f0de800bd70 comp rx=0 tx=0).stop 2024-03-24T09:41:01.837 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.836+0000 7f0df342d700 1 -- 172.21.15.155:0/1092857587 shutdown_connections 2024-03-24T09:41:01.837 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.836+0000 7f0df342d700 1 --2- 172.21.15.155:0/1092857587 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f0dd8038810 0x7f0dd803acd0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:01.838 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.836+0000 7f0df342d700 1 --2- 172.21.15.155:0/1092857587 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0dec0ff640 0x7f0dec194ae0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:01.838 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.836+0000 7f0df342d700 1 -- 172.21.15.155:0/1092857587 >> 172.21.15.155:0/1092857587 conn(0x7f0dec0fabe0 msgr2=0x7f0dec0fb670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:41:01.838 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.836+0000 7f0df342d700 1 -- 172.21.15.155:0/1092857587 shutdown_connections 2024-03-24T09:41:01.838 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:01.836+0000 7f0df342d700 1 -- 172.21.15.155:0/1092857587 wait complete. 2024-03-24T09:41:02.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:02 smithi022 ceph-mon[29512]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:41:02.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:02 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:02.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:02 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:02.454 DEBUG:teuthology.orchestra.run.smithi155:mon.smithi155> sudo journalctl -f -n 0 -u ceph-45356448-e9c2-11ee-95cd-87774f69a715@mon.smithi155.service 2024-03-24T09:41:02.457 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T09:41:02.457 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T09:41:02.684 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T09:41:02.730 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T09:41:03.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:03 smithi022 ceph-mon[29512]: from='client.14197 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi022:172.21.15.22=smithi022;smithi155:172.21.15.155=smithi155", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:41:03.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:03 smithi022 ceph-mon[29512]: Saving service mon spec with placement smithi022:172.21.15.22=smithi022;smithi155:172.21.15.155=smithi155;count:2 2024-03-24T09:41:04.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.139+0000 7f84b7f4c700 1 -- 172.21.15.155:0/930631782 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f84b0101250 msgr2=0x7f84b0103640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:04.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.139+0000 7f84b7f4c700 1 --2- 172.21.15.155:0/930631782 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f84b0101250 0x7f84b0103640 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f84a0009910 tx=0x7f84a0009c20 comp rx=0 tx=0).stop 2024-03-24T09:41:04.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.139+0000 7f84b7f4c700 1 -- 172.21.15.155:0/930631782 shutdown_connections 2024-03-24T09:41:04.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.139+0000 7f84b7f4c700 1 --2- 172.21.15.155:0/930631782 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f84b0101250 0x7f84b0103640 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:04.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.139+0000 7f84b7f4c700 1 -- 172.21.15.155:0/930631782 >> 172.21.15.155:0/930631782 conn(0x7f84b00fabe0 msgr2=0x7f84b00fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:41:04.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.139+0000 7f84b7f4c700 1 -- 172.21.15.155:0/930631782 shutdown_connections 2024-03-24T09:41:04.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.139+0000 7f84b7f4c700 1 -- 172.21.15.155:0/930631782 wait complete. 2024-03-24T09:41:04.143 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.140+0000 7f84b7f4c700 1 Processor -- start 2024-03-24T09:41:04.143 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.140+0000 7f84b7f4c700 1 -- start start 2024-03-24T09:41:04.143 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.140+0000 7f84b7f4c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f84b0101250 0x7f84b0100a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:41:04.143 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.140+0000 7f84b7f4c700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b0100fc0 con 0x7f84b0101250 2024-03-24T09:41:04.143 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.140+0000 7f84b5ce8700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f84b0101250 0x7f84b0100a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:41:04.143 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.140+0000 7f84b5ce8700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f84b0101250 0x7f84b0100a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.155:56728/0 (socket says 172.21.15.155:56728) 2024-03-24T09:41:04.143 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.140+0000 7f84b5ce8700 1 -- 172.21.15.155:0/2563688890 learned_addr learned my addr 172.21.15.155:0/2563688890 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-24T09:41:04.143 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.141+0000 7f84b5ce8700 1 -- 172.21.15.155:0/2563688890 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a00095f0 con 0x7f84b0101250 2024-03-24T09:41:04.143 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.141+0000 7f84b5ce8700 1 --2- 172.21.15.155:0/2563688890 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f84b0101250 0x7f84b0100a80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f84a0009ed0 tx=0x7f84a0004690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:41:04.143 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.141+0000 7f84a6ffd700 1 -- 172.21.15.155:0/2563688890 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84a001d070 con 0x7f84b0101250 2024-03-24T09:41:04.143 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.141+0000 7f84b7f4c700 1 -- 172.21.15.155:0/2563688890 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84b00ff200 con 0x7f84b0101250 2024-03-24T09:41:04.143 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.141+0000 7f84a6ffd700 1 -- 172.21.15.155:0/2563688890 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f84a000be20 con 0x7f84b0101250 2024-03-24T09:41:04.143 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.141+0000 7f84a6ffd700 1 -- 172.21.15.155:0/2563688890 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84a000f460 con 0x7f84b0101250 2024-03-24T09:41:04.143 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.141+0000 7f84b7f4c700 1 -- 172.21.15.155:0/2563688890 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84b00ff6a0 con 0x7f84b0101250 2024-03-24T09:41:04.144 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.142+0000 7f84a6ffd700 1 -- 172.21.15.155:0/2563688890 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 13) v1 ==== 45604+0+0 (secure 0 0 0) 0x7f84a000f650 con 0x7f84b0101250 2024-03-24T09:41:04.144 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.142+0000 7f84b7f4c700 1 -- 172.21.15.155:0/2563688890 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84b0106ea0 con 0x7f84b0101250 2024-03-24T09:41:04.144 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.142+0000 7f84a6ffd700 1 --2- 172.21.15.155:0/2563688890 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f849c0387d0 0x7f849c03ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:41:04.144 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.142+0000 7f84a6ffd700 1 -- 172.21.15.155:0/2563688890 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f84a004cb40 con 0x7f84b0101250 2024-03-24T09:41:04.144 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.142+0000 7f84b54e7700 1 --2- 172.21.15.155:0/2563688890 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f849c0387d0 0x7f849c03ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:41:04.145 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.143+0000 7f84b54e7700 1 --2- 172.21.15.155:0/2563688890 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f849c0387d0 0x7f849c03ac90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f84ac006fd0 tx=0x7f84ac006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:41:04.148 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.148+0000 7f84a6ffd700 1 -- 172.21.15.155:0/2563688890 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f84a002a360 con 0x7f84b0101250 2024-03-24T09:41:04.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:04 smithi022 ceph-mon[29512]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:41:04.419 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.418+0000 7f84b7f4c700 1 -- 172.21.15.155:0/2563688890 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f84b00623e0 con 0x7f84b0101250 2024-03-24T09:41:04.420 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.419+0000 7f84a6ffd700 1 -- 172.21.15.155:0/2563688890 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f84a0020b60 con 0x7f84b0101250 2024-03-24T09:41:04.420 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:41:04.420 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"45356448-e9c2-11ee-95cd-87774f69a715","modified":"2024-03-24T09:39:22.550742Z","created":"2024-03-24T09:39:22.550742Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi022","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:3300","nonce":0},{"type":"v1","addr":"172.21.15.22:6789","nonce":0}]},"addr":"172.21.15.22:6789/0","public_addr":"172.21.15.22:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T09:41:04.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.421+0000 7f84b7f4c700 1 -- 172.21.15.155:0/2563688890 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f849c0387d0 msgr2=0x7f849c03ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:04.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.421+0000 7f84b7f4c700 1 --2- 172.21.15.155:0/2563688890 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f849c0387d0 0x7f849c03ac90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f84ac006fd0 tx=0x7f84ac006e40 comp rx=0 tx=0).stop 2024-03-24T09:41:04.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.421+0000 7f84b7f4c700 1 -- 172.21.15.155:0/2563688890 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f84b0101250 msgr2=0x7f84b0100a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:04.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.421+0000 7f84b7f4c700 1 --2- 172.21.15.155:0/2563688890 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f84b0101250 0x7f84b0100a80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f84a0009ed0 tx=0x7f84a0004690 comp rx=0 tx=0).stop 2024-03-24T09:41:04.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.421+0000 7f84b7f4c700 1 -- 172.21.15.155:0/2563688890 shutdown_connections 2024-03-24T09:41:04.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.421+0000 7f84b7f4c700 1 --2- 172.21.15.155:0/2563688890 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f849c0387d0 0x7f849c03ac90 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:04.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.421+0000 7f84b7f4c700 1 --2- 172.21.15.155:0/2563688890 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f84b0101250 0x7f84b0100a80 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:04.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.421+0000 7f84b7f4c700 1 -- 172.21.15.155:0/2563688890 >> 172.21.15.155:0/2563688890 conn(0x7f84b00fabe0 msgr2=0x7f84b00fb650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:41:04.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.421+0000 7f84b7f4c700 1 -- 172.21.15.155:0/2563688890 shutdown_connections 2024-03-24T09:41:04.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:04.421+0000 7f84b7f4c700 1 -- 172.21.15.155:0/2563688890 wait complete. 2024-03-24T09:41:04.423 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-03-24T09:41:05.169 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:05 smithi022 ceph-mon[29512]: from='client.? 172.21.15.155:0/2563688890' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T09:41:05.948 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T09:41:05.948 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T09:41:06.145 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T09:41:06.191 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T09:41:06.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:06 smithi022 ceph-mon[29512]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:41:06.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:06 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:06.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:06 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:06.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:06 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:06.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:06 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:06.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:06 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:06.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:06 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:06.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:06 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:06.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:06 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:06.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:06 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T09:41:06.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:06 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:06.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:06 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:07.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:07 smithi022 ceph-mon[29512]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-24T09:41:07.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:07 smithi022 ceph-mon[29512]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T09:41:07.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:07 smithi022 ceph-mon[29512]: Deploying daemon grafana.smithi022 on smithi022 2024-03-24T09:41:07.612 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.611+0000 7f15fd191700 1 -- 172.21.15.155:0/2777083065 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f15f8101d10 msgr2=0x7f15f8102130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:07.612 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.611+0000 7f15fd191700 1 --2- 172.21.15.155:0/2777083065 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f15f8101d10 0x7f15f8102130 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f15ec009910 tx=0x7f15ec009c20 comp rx=0 tx=0).stop 2024-03-24T09:41:07.612 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.611+0000 7f15fd191700 1 -- 172.21.15.155:0/2777083065 shutdown_connections 2024-03-24T09:41:07.612 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.611+0000 7f15fd191700 1 --2- 172.21.15.155:0/2777083065 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f15f8101d10 0x7f15f8102130 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:07.612 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.611+0000 7f15fd191700 1 -- 172.21.15.155:0/2777083065 >> 172.21.15.155:0/2777083065 conn(0x7f15f80fa370 msgr2=0x7f15f80fc790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:41:07.612 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.611+0000 7f15fd191700 1 -- 172.21.15.155:0/2777083065 shutdown_connections 2024-03-24T09:41:07.612 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.611+0000 7f15fd191700 1 -- 172.21.15.155:0/2777083065 wait complete. 2024-03-24T09:41:07.612 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.612+0000 7f15fd191700 1 Processor -- start 2024-03-24T09:41:07.613 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.612+0000 7f15fd191700 1 -- start start 2024-03-24T09:41:07.613 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.612+0000 7f15fd191700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f15f8101d10 0x7f15f8195be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:41:07.613 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.612+0000 7f15fd191700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f8196120 con 0x7f15f8101d10 2024-03-24T09:41:07.614 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.613+0000 7f15f6d9d700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f15f8101d10 0x7f15f8195be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:41:07.614 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.613+0000 7f15f6d9d700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f15f8101d10 0x7f15f8195be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.155:59546/0 (socket says 172.21.15.155:59546) 2024-03-24T09:41:07.614 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.613+0000 7f15f6d9d700 1 -- 172.21.15.155:0/2296248688 learned_addr learned my addr 172.21.15.155:0/2296248688 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-24T09:41:07.614 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.613+0000 7f15f6d9d700 1 -- 172.21.15.155:0/2296248688 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15ec0095f0 con 0x7f15f8101d10 2024-03-24T09:41:07.614 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.613+0000 7f15f6d9d700 1 --2- 172.21.15.155:0/2296248688 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f15f8101d10 0x7f15f8195be0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f15ec006010 tx=0x7f15ec009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:41:07.614 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.613+0000 7f15dffff700 1 -- 172.21.15.155:0/2296248688 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15ec01d070 con 0x7f15f8101d10 2024-03-24T09:41:07.615 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.613+0000 7f15fd191700 1 -- 172.21.15.155:0/2296248688 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15f8196320 con 0x7f15f8101d10 2024-03-24T09:41:07.615 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.613+0000 7f15dffff700 1 -- 172.21.15.155:0/2296248688 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f15ec005dc0 con 0x7f15f8101d10 2024-03-24T09:41:07.615 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.613+0000 7f15dffff700 1 -- 172.21.15.155:0/2296248688 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15ec022e90 con 0x7f15f8101d10 2024-03-24T09:41:07.615 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.613+0000 7f15fd191700 1 -- 172.21.15.155:0/2296248688 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15f81967c0 con 0x7f15f8101d10 2024-03-24T09:41:07.615 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.614+0000 7f15dffff700 1 -- 172.21.15.155:0/2296248688 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 13) v1 ==== 45604+0+0 (secure 0 0 0) 0x7f15ec00b960 con 0x7f15f8101d10 2024-03-24T09:41:07.615 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.614+0000 7f15fd191700 1 -- 172.21.15.155:0/2296248688 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15f818f560 con 0x7f15f8101d10 2024-03-24T09:41:07.615 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.614+0000 7f15dffff700 1 --2- 172.21.15.155:0/2296248688 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f15e0038760 0x7f15e003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:41:07.615 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.615+0000 7f15dffff700 1 -- 172.21.15.155:0/2296248688 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f15ec04c8c0 con 0x7f15f8101d10 2024-03-24T09:41:07.615 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.615+0000 7f15f659c700 1 --2- 172.21.15.155:0/2296248688 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f15e0038760 0x7f15e003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:41:07.615 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.615+0000 7f15f659c700 1 --2- 172.21.15.155:0/2296248688 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f15e0038760 0x7f15e003ac20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f15e8006fd0 tx=0x7f15e8006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:41:07.620 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.621+0000 7f15dffff700 1 -- 172.21.15.155:0/2296248688 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f15ec022750 con 0x7f15f8101d10 2024-03-24T09:41:07.889 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.889+0000 7f15fd191700 1 -- 172.21.15.155:0/2296248688 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f15f80623e0 con 0x7f15f8101d10 2024-03-24T09:41:07.889 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.889+0000 7f15dffff700 1 -- 172.21.15.155:0/2296248688 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f15ec027020 con 0x7f15f8101d10 2024-03-24T09:41:07.889 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:41:07.889 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"45356448-e9c2-11ee-95cd-87774f69a715","modified":"2024-03-24T09:39:22.550742Z","created":"2024-03-24T09:39:22.550742Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi022","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:3300","nonce":0},{"type":"v1","addr":"172.21.15.22:6789","nonce":0}]},"addr":"172.21.15.22:6789/0","public_addr":"172.21.15.22:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T09:41:07.891 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.891+0000 7f15fd191700 1 -- 172.21.15.155:0/2296248688 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f15e0038760 msgr2=0x7f15e003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:07.891 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.891+0000 7f15fd191700 1 --2- 172.21.15.155:0/2296248688 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f15e0038760 0x7f15e003ac20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f15e8006fd0 tx=0x7f15e8006e40 comp rx=0 tx=0).stop 2024-03-24T09:41:07.891 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.891+0000 7f15fd191700 1 -- 172.21.15.155:0/2296248688 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f15f8101d10 msgr2=0x7f15f8195be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:07.891 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.891+0000 7f15fd191700 1 --2- 172.21.15.155:0/2296248688 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f15f8101d10 0x7f15f8195be0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f15ec006010 tx=0x7f15ec009f80 comp rx=0 tx=0).stop 2024-03-24T09:41:07.892 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.891+0000 7f15fd191700 1 -- 172.21.15.155:0/2296248688 shutdown_connections 2024-03-24T09:41:07.892 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.891+0000 7f15fd191700 1 --2- 172.21.15.155:0/2296248688 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f15e0038760 0x7f15e003ac20 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:07.892 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.891+0000 7f15fd191700 1 --2- 172.21.15.155:0/2296248688 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f15f8101d10 0x7f15f8195be0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:07.892 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.891+0000 7f15fd191700 1 -- 172.21.15.155:0/2296248688 >> 172.21.15.155:0/2296248688 conn(0x7f15f80fa370 msgr2=0x7f15f80fae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:41:07.892 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.892+0000 7f15fd191700 1 -- 172.21.15.155:0/2296248688 shutdown_connections 2024-03-24T09:41:07.892 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:07.892+0000 7f15fd191700 1 -- 172.21.15.155:0/2296248688 wait complete. 2024-03-24T09:41:07.892 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-03-24T09:41:08.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:08 smithi022 ceph-mon[29512]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:41:08.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:08 smithi022 ceph-mon[29512]: from='client.? 172.21.15.155:0/2296248688' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T09:41:09.554 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T09:41:09.554 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T09:41:09.757 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T09:41:09.802 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T09:41:10.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:10 smithi022 ceph-mon[29512]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:41:11.275 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.274+0000 7f30f7136700 1 -- 172.21.15.155:0/4066835434 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f30f0101240 msgr2=0x7f30f0103630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:11.275 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.274+0000 7f30f7136700 1 --2- 172.21.15.155:0/4066835434 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f30f0101240 0x7f30f0103630 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f30e4009910 tx=0x7f30e4009c20 comp rx=0 tx=0).stop 2024-03-24T09:41:11.276 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.274+0000 7f30f7136700 1 -- 172.21.15.155:0/4066835434 shutdown_connections 2024-03-24T09:41:11.276 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.274+0000 7f30f7136700 1 --2- 172.21.15.155:0/4066835434 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f30f0101240 0x7f30f0103630 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:11.276 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.274+0000 7f30f7136700 1 -- 172.21.15.155:0/4066835434 >> 172.21.15.155:0/4066835434 conn(0x7f30f00fabd0 msgr2=0x7f30f00fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:41:11.276 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.274+0000 7f30f7136700 1 -- 172.21.15.155:0/4066835434 shutdown_connections 2024-03-24T09:41:11.276 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.274+0000 7f30f7136700 1 -- 172.21.15.155:0/4066835434 wait complete. 2024-03-24T09:41:11.276 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.274+0000 7f30f7136700 1 Processor -- start 2024-03-24T09:41:11.276 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.274+0000 7f30f7136700 1 -- start start 2024-03-24T09:41:11.276 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.275+0000 7f30f7136700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f30f0101240 0x7f30f0195bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:41:11.276 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.275+0000 7f30f7136700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f0196110 con 0x7f30f0101240 2024-03-24T09:41:11.276 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.275+0000 7f30f4ed2700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f30f0101240 0x7f30f0195bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:41:11.276 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.275+0000 7f30f4ed2700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f30f0101240 0x7f30f0195bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.155:59558/0 (socket says 172.21.15.155:59558) 2024-03-24T09:41:11.276 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.275+0000 7f30f4ed2700 1 -- 172.21.15.155:0/2838244613 learned_addr learned my addr 172.21.15.155:0/2838244613 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-24T09:41:11.276 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.275+0000 7f30f4ed2700 1 -- 172.21.15.155:0/2838244613 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e40095f0 con 0x7f30f0101240 2024-03-24T09:41:11.277 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.275+0000 7f30f4ed2700 1 --2- 172.21.15.155:0/2838244613 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f30f0101240 0x7f30f0195bd0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f30e4009ed0 tx=0x7f30e4004690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:41:11.277 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.275+0000 7f30edffb700 1 -- 172.21.15.155:0/2838244613 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30e401d070 con 0x7f30f0101240 2024-03-24T09:41:11.277 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.275+0000 7f30edffb700 1 -- 172.21.15.155:0/2838244613 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f30e400be20 con 0x7f30f0101240 2024-03-24T09:41:11.277 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.275+0000 7f30f7136700 1 -- 172.21.15.155:0/2838244613 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30f0196310 con 0x7f30f0101240 2024-03-24T09:41:11.277 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.275+0000 7f30edffb700 1 -- 172.21.15.155:0/2838244613 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30e400f460 con 0x7f30f0101240 2024-03-24T09:41:11.277 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.276+0000 7f30f7136700 1 -- 172.21.15.155:0/2838244613 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30f01967b0 con 0x7f30f0101240 2024-03-24T09:41:11.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.276+0000 7f30edffb700 1 -- 172.21.15.155:0/2838244613 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 13) v1 ==== 45604+0+0 (secure 0 0 0) 0x7f30e400b960 con 0x7f30f0101240 2024-03-24T09:41:11.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.276+0000 7f30f7136700 1 -- 172.21.15.155:0/2838244613 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30f018f560 con 0x7f30f0101240 2024-03-24T09:41:11.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.277+0000 7f30edffb700 1 --2- 172.21.15.155:0/2838244613 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f30e0038820 0x7f30e003ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:41:11.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.277+0000 7f30edffb700 1 -- 172.21.15.155:0/2838244613 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f30e404c950 con 0x7f30f0101240 2024-03-24T09:41:11.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.277+0000 7f30effff700 1 --2- 172.21.15.155:0/2838244613 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f30e0038820 0x7f30e003ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:41:11.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.277+0000 7f30effff700 1 --2- 172.21.15.155:0/2838244613 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f30e0038820 0x7f30e003ace0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f30dc006fd0 tx=0x7f30dc006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:41:11.281 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.282+0000 7f30edffb700 1 -- 172.21.15.155:0/2838244613 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f30e402a360 con 0x7f30f0101240 2024-03-24T09:41:11.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:11 smithi022 ceph-mon[29512]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:41:11.549 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.549+0000 7f30f7136700 1 -- 172.21.15.155:0/2838244613 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f30f00623e0 con 0x7f30f0101240 2024-03-24T09:41:11.549 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.550+0000 7f30edffb700 1 -- 172.21.15.155:0/2838244613 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f30f00623e0 con 0x7f30f0101240 2024-03-24T09:41:11.549 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:41:11.550 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"45356448-e9c2-11ee-95cd-87774f69a715","modified":"2024-03-24T09:39:22.550742Z","created":"2024-03-24T09:39:22.550742Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi022","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:3300","nonce":0},{"type":"v1","addr":"172.21.15.22:6789","nonce":0}]},"addr":"172.21.15.22:6789/0","public_addr":"172.21.15.22:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T09:41:11.552 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.552+0000 7f30f7136700 1 -- 172.21.15.155:0/2838244613 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f30e0038820 msgr2=0x7f30e003ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:11.552 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.552+0000 7f30f7136700 1 --2- 172.21.15.155:0/2838244613 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f30e0038820 0x7f30e003ace0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f30dc006fd0 tx=0x7f30dc006e40 comp rx=0 tx=0).stop 2024-03-24T09:41:11.552 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.552+0000 7f30f7136700 1 -- 172.21.15.155:0/2838244613 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f30f0101240 msgr2=0x7f30f0195bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:11.552 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.552+0000 7f30f7136700 1 --2- 172.21.15.155:0/2838244613 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f30f0101240 0x7f30f0195bd0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f30e4009ed0 tx=0x7f30e4004690 comp rx=0 tx=0).stop 2024-03-24T09:41:11.552 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.552+0000 7f30f7136700 1 -- 172.21.15.155:0/2838244613 shutdown_connections 2024-03-24T09:41:11.552 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.552+0000 7f30f7136700 1 --2- 172.21.15.155:0/2838244613 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f30e0038820 0x7f30e003ace0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:11.552 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.552+0000 7f30f7136700 1 --2- 172.21.15.155:0/2838244613 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f30f0101240 0x7f30f0195bd0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:11.552 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.552+0000 7f30f7136700 1 -- 172.21.15.155:0/2838244613 >> 172.21.15.155:0/2838244613 conn(0x7f30f00fabd0 msgr2=0x7f30f00fb6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:41:11.552 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.552+0000 7f30f7136700 1 -- 172.21.15.155:0/2838244613 shutdown_connections 2024-03-24T09:41:11.552 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:11.552+0000 7f30f7136700 1 -- 172.21.15.155:0/2838244613 wait complete. 2024-03-24T09:41:11.553 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-03-24T09:41:12.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:12 smithi022 ceph-mon[29512]: from='client.? 172.21.15.155:0/2838244613' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T09:41:13.133 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T09:41:13.133 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T09:41:13.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:13 smithi022 ceph-mon[29512]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:41:13.333 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T09:41:13.379 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T09:41:14.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:14.833+0000 7f7ca1742700 1 -- 172.21.15.155:0/4216244440 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7c9c0ff640 msgr2=0x7f7c9c0ffa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:14.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:14.833+0000 7f7ca1742700 1 --2- 172.21.15.155:0/4216244440 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7c9c0ff640 0x7f7c9c0ffa60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c009910 tx=0x7f7c8c009c20 comp rx=0 tx=0).stop 2024-03-24T09:41:14.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:14.833+0000 7f7ca1742700 1 -- 172.21.15.155:0/4216244440 shutdown_connections 2024-03-24T09:41:14.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:14.833+0000 7f7ca1742700 1 --2- 172.21.15.155:0/4216244440 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7c9c0ff640 0x7f7c9c0ffa60 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:14.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:14.833+0000 7f7ca1742700 1 -- 172.21.15.155:0/4216244440 >> 172.21.15.155:0/4216244440 conn(0x7f7c9c0fabe0 msgr2=0x7f7c9c0fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:41:14.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:14.834+0000 7f7ca1742700 1 -- 172.21.15.155:0/4216244440 shutdown_connections 2024-03-24T09:41:14.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:14.834+0000 7f7ca1742700 1 -- 172.21.15.155:0/4216244440 wait complete. 2024-03-24T09:41:14.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:14.834+0000 7f7ca1742700 1 Processor -- start 2024-03-24T09:41:14.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:14.834+0000 7f7ca1742700 1 -- start start 2024-03-24T09:41:14.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:14.834+0000 7f7ca1742700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7c9c0ff640 0x7f7c9c068940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:41:14.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:14.834+0000 7f7ca1742700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9c068e80 con 0x7f7c9c0ff640 2024-03-24T09:41:14.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:14.835+0000 7f7c9affd700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7c9c0ff640 0x7f7c9c068940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:41:14.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:14.835+0000 7f7c9affd700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7c9c0ff640 0x7f7c9c068940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.155:59572/0 (socket says 172.21.15.155:59572) 2024-03-24T09:41:14.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:14.835+0000 7f7c9affd700 1 -- 172.21.15.155:0/2772574405 learned_addr learned my addr 172.21.15.155:0/2772574405 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-24T09:41:14.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:14.835+0000 7f7c9affd700 1 -- 172.21.15.155:0/2772574405 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c8c0095f0 con 0x7f7c9c0ff640 2024-03-24T09:41:14.837 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:14.835+0000 7f7c9affd700 1 --2- 172.21.15.155:0/2772574405 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7c9c0ff640 0x7f7c9c068940 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c006010 tx=0x7f7c8c0050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:41:14.837 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:14.835+0000 7f7c98ff9700 1 -- 172.21.15.155:0/2772574405 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c8c01d070 con 0x7f7c9c0ff640 2024-03-24T09:41:14.837 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:14.835+0000 7f7ca1742700 1 -- 172.21.15.155:0/2772574405 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c9c198910 con 0x7f7c9c0ff640 2024-03-24T09:41:14.837 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:14.835+0000 7f7c98ff9700 1 -- 172.21.15.155:0/2772574405 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7c8c00be20 con 0x7f7c9c0ff640 2024-03-24T09:41:14.837 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:14.835+0000 7f7ca1742700 1 -- 172.21.15.155:0/2772574405 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c9c198db0 con 0x7f7c9c0ff640 2024-03-24T09:41:14.837 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:14.835+0000 7f7c98ff9700 1 -- 172.21.15.155:0/2772574405 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c8c022e40 con 0x7f7c9c0ff640 2024-03-24T09:41:14.838 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:14.836+0000 7f7c98ff9700 1 -- 172.21.15.155:0/2772574405 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 13) v1 ==== 45604+0+0 (secure 0 0 0) 0x7f7c8c00b960 con 0x7f7c9c0ff640 2024-03-24T09:41:14.838 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:14.836+0000 7f7ca1742700 1 -- 172.21.15.155:0/2772574405 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c9c18e470 con 0x7f7c9c0ff640 2024-03-24T09:41:14.838 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:14.836+0000 7f7c98ff9700 1 --2- 172.21.15.155:0/2772574405 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f7c880387d0 0x7f7c8803ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:41:14.838 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:14.836+0000 7f7c98ff9700 1 -- 172.21.15.155:0/2772574405 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7c8c04caa0 con 0x7f7c9c0ff640 2024-03-24T09:41:14.838 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:14.836+0000 7f7c927fc700 1 --2- 172.21.15.155:0/2772574405 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f7c880387d0 0x7f7c8803ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:41:14.838 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:14.837+0000 7f7c927fc700 1 --2- 172.21.15.155:0/2772574405 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f7c880387d0 0x7f7c8803ac90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7c84006fd0 tx=0x7f7c84006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:41:14.845 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:14.846+0000 7f7c98ff9700 1 -- 172.21.15.155:0/2772574405 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7c8c022700 con 0x7f7c9c0ff640 2024-03-24T09:41:15.113 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:15.113+0000 7f7ca1742700 1 -- 172.21.15.155:0/2772574405 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7c9c02d110 con 0x7f7c9c0ff640 2024-03-24T09:41:15.113 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:15.114+0000 7f7c98ff9700 1 -- 172.21.15.155:0/2772574405 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f7c8c027030 con 0x7f7c9c0ff640 2024-03-24T09:41:15.113 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:41:15.114 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"45356448-e9c2-11ee-95cd-87774f69a715","modified":"2024-03-24T09:39:22.550742Z","created":"2024-03-24T09:39:22.550742Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi022","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:3300","nonce":0},{"type":"v1","addr":"172.21.15.22:6789","nonce":0}]},"addr":"172.21.15.22:6789/0","public_addr":"172.21.15.22:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T09:41:15.115 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:15.116+0000 7f7ca1742700 1 -- 172.21.15.155:0/2772574405 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f7c880387d0 msgr2=0x7f7c8803ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:15.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:15.116+0000 7f7ca1742700 1 --2- 172.21.15.155:0/2772574405 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f7c880387d0 0x7f7c8803ac90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7c84006fd0 tx=0x7f7c84006e40 comp rx=0 tx=0).stop 2024-03-24T09:41:15.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:15.116+0000 7f7ca1742700 1 -- 172.21.15.155:0/2772574405 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7c9c0ff640 msgr2=0x7f7c9c068940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:15.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:15.116+0000 7f7ca1742700 1 --2- 172.21.15.155:0/2772574405 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7c9c0ff640 0x7f7c9c068940 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c006010 tx=0x7f7c8c0050b0 comp rx=0 tx=0).stop 2024-03-24T09:41:15.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:15.116+0000 7f7ca1742700 1 -- 172.21.15.155:0/2772574405 shutdown_connections 2024-03-24T09:41:15.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:15.116+0000 7f7ca1742700 1 --2- 172.21.15.155:0/2772574405 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f7c880387d0 0x7f7c8803ac90 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:15.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:15.116+0000 7f7ca1742700 1 --2- 172.21.15.155:0/2772574405 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7c9c0ff640 0x7f7c9c068940 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:15.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:15.116+0000 7f7ca1742700 1 -- 172.21.15.155:0/2772574405 >> 172.21.15.155:0/2772574405 conn(0x7f7c9c0fabe0 msgr2=0x7f7c9c0fb670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:41:15.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:15.116+0000 7f7ca1742700 1 -- 172.21.15.155:0/2772574405 shutdown_connections 2024-03-24T09:41:15.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:15.116+0000 7f7ca1742700 1 -- 172.21.15.155:0/2772574405 wait complete. 2024-03-24T09:41:15.116 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-03-24T09:41:16.270 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:16 smithi022 ceph-mon[29512]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:41:16.270 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:16 smithi022 ceph-mon[29512]: from='client.? 172.21.15.155:0/2772574405' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T09:41:16.737 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T09:41:16.738 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T09:41:16.939 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T09:41:16.984 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T09:41:18.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:18 smithi022 ceph-mon[29512]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:41:18.548 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.548+0000 7fb2e4840700 1 -- 172.21.15.155:0/3547600049 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb2e00ff630 msgr2=0x7fb2e00ffa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:18.548 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.548+0000 7fb2e4840700 1 --2- 172.21.15.155:0/3547600049 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb2e00ff630 0x7fb2e00ffa50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0009910 tx=0x7fb2d0009c20 comp rx=0 tx=0).stop 2024-03-24T09:41:18.548 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.548+0000 7fb2e4840700 1 -- 172.21.15.155:0/3547600049 shutdown_connections 2024-03-24T09:41:18.548 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.548+0000 7fb2e4840700 1 --2- 172.21.15.155:0/3547600049 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb2e00ff630 0x7fb2e00ffa50 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:18.548 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.548+0000 7fb2e4840700 1 -- 172.21.15.155:0/3547600049 >> 172.21.15.155:0/3547600049 conn(0x7fb2e00fabd0 msgr2=0x7fb2e00fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:41:18.548 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.548+0000 7fb2e4840700 1 -- 172.21.15.155:0/3547600049 shutdown_connections 2024-03-24T09:41:18.548 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.548+0000 7fb2e4840700 1 -- 172.21.15.155:0/3547600049 wait complete. 2024-03-24T09:41:18.550 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.549+0000 7fb2e4840700 1 Processor -- start 2024-03-24T09:41:18.550 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.549+0000 7fb2e4840700 1 -- start start 2024-03-24T09:41:18.550 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.549+0000 7fb2e4840700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb2e00ff630 0x7fb2e0069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:41:18.550 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.549+0000 7fb2e4840700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e0069a70 con 0x7fb2e00ff630 2024-03-24T09:41:18.550 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.549+0000 7fb2de59c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb2e00ff630 0x7fb2e0069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:41:18.550 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.549+0000 7fb2de59c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb2e00ff630 0x7fb2e0069530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.155:35656/0 (socket says 172.21.15.155:35656) 2024-03-24T09:41:18.550 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.549+0000 7fb2de59c700 1 -- 172.21.15.155:0/3194827410 learned_addr learned my addr 172.21.15.155:0/3194827410 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-24T09:41:18.550 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.549+0000 7fb2de59c700 1 -- 172.21.15.155:0/3194827410 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d00095f0 con 0x7fb2e00ff630 2024-03-24T09:41:18.550 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.550+0000 7fb2de59c700 1 --2- 172.21.15.155:0/3194827410 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb2e00ff630 0x7fb2e0069530 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0006010 tx=0x7fb2d0009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:41:18.550 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.550+0000 7fb2cf7fe700 1 -- 172.21.15.155:0/3194827410 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2d001d070 con 0x7fb2e00ff630 2024-03-24T09:41:18.550 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.550+0000 7fb2e4840700 1 -- 172.21.15.155:0/3194827410 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2e0198950 con 0x7fb2e00ff630 2024-03-24T09:41:18.550 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.550+0000 7fb2cf7fe700 1 -- 172.21.15.155:0/3194827410 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb2d0005dc0 con 0x7fb2e00ff630 2024-03-24T09:41:18.551 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.550+0000 7fb2cf7fe700 1 -- 172.21.15.155:0/3194827410 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2d0022e90 con 0x7fb2e00ff630 2024-03-24T09:41:18.551 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.550+0000 7fb2e4840700 1 -- 172.21.15.155:0/3194827410 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2e0198d90 con 0x7fb2e00ff630 2024-03-24T09:41:18.551 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.551+0000 7fb2cf7fe700 1 -- 172.21.15.155:0/3194827410 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 13) v1 ==== 45604+0+0 (secure 0 0 0) 0x7fb2d000f4b0 con 0x7fb2e00ff630 2024-03-24T09:41:18.552 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.551+0000 7fb2e4840700 1 -- 172.21.15.155:0/3194827410 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2e018e460 con 0x7fb2e00ff630 2024-03-24T09:41:18.552 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.551+0000 7fb2cf7fe700 1 --2- 172.21.15.155:0/3194827410 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7fb2c80387c0 0x7fb2c803ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:41:18.552 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.551+0000 7fb2cf7fe700 1 -- 172.21.15.155:0/3194827410 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb2d004c8d0 con 0x7fb2e00ff630 2024-03-24T09:41:18.552 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.551+0000 7fb2ddd9b700 1 --2- 172.21.15.155:0/3194827410 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7fb2c80387c0 0x7fb2c803ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:41:18.552 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.552+0000 7fb2ddd9b700 1 --2- 172.21.15.155:0/3194827410 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7fb2c80387c0 0x7fb2c803ac80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4006fd0 tx=0x7fb2d4006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:41:18.556 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.556+0000 7fb2cf7fe700 1 -- 172.21.15.155:0/3194827410 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb2d0017440 con 0x7fb2e00ff630 2024-03-24T09:41:18.823 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.824+0000 7fb2e4840700 1 -- 172.21.15.155:0/3194827410 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb2e00623e0 con 0x7fb2e00ff630 2024-03-24T09:41:18.824 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.824+0000 7fb2cf7fe700 1 -- 172.21.15.155:0/3194827410 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fb2d0027020 con 0x7fb2e00ff630 2024-03-24T09:41:18.824 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:41:18.824 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"45356448-e9c2-11ee-95cd-87774f69a715","modified":"2024-03-24T09:39:22.550742Z","created":"2024-03-24T09:39:22.550742Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi022","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:3300","nonce":0},{"type":"v1","addr":"172.21.15.22:6789","nonce":0}]},"addr":"172.21.15.22:6789/0","public_addr":"172.21.15.22:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T09:41:18.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.826+0000 7fb2e4840700 1 -- 172.21.15.155:0/3194827410 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7fb2c80387c0 msgr2=0x7fb2c803ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:18.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.826+0000 7fb2e4840700 1 --2- 172.21.15.155:0/3194827410 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7fb2c80387c0 0x7fb2c803ac80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4006fd0 tx=0x7fb2d4006e40 comp rx=0 tx=0).stop 2024-03-24T09:41:18.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.826+0000 7fb2e4840700 1 -- 172.21.15.155:0/3194827410 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb2e00ff630 msgr2=0x7fb2e0069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:18.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.826+0000 7fb2e4840700 1 --2- 172.21.15.155:0/3194827410 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb2e00ff630 0x7fb2e0069530 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0006010 tx=0x7fb2d0009f80 comp rx=0 tx=0).stop 2024-03-24T09:41:18.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.826+0000 7fb2e4840700 1 -- 172.21.15.155:0/3194827410 shutdown_connections 2024-03-24T09:41:18.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.826+0000 7fb2e4840700 1 --2- 172.21.15.155:0/3194827410 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7fb2c80387c0 0x7fb2c803ac80 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:18.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.826+0000 7fb2e4840700 1 --2- 172.21.15.155:0/3194827410 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb2e00ff630 0x7fb2e0069530 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:18.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.826+0000 7fb2e4840700 1 -- 172.21.15.155:0/3194827410 >> 172.21.15.155:0/3194827410 conn(0x7fb2e00fabd0 msgr2=0x7fb2e00fb6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:41:18.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.826+0000 7fb2e4840700 1 -- 172.21.15.155:0/3194827410 shutdown_connections 2024-03-24T09:41:18.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:18.827+0000 7fb2e4840700 1 -- 172.21.15.155:0/3194827410 wait complete. 2024-03-24T09:41:18.827 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-03-24T09:41:19.275 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:19 smithi022 ceph-mon[29512]: from='client.? 172.21.15.155:0/3194827410' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T09:41:20.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:20 smithi022 ceph-mon[29512]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:41:20.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:20 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:20.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:20 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:20.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:20 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:20.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:20 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:20.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:20 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:20.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:20 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:20.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:20 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:20.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:20 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:20.416 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T09:41:20.416 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T09:41:20.618 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T09:41:20.664 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T09:41:21.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:21 smithi022 ceph-mon[29512]: Deploying daemon prometheus.smithi022 on smithi022 2024-03-24T09:41:22.152 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.150+0000 7f10e1d71700 1 -- 172.21.15.155:0/3337551771 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f10dc0ff630 msgr2=0x7f10dc0ffa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:22.152 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.150+0000 7f10e1d71700 1 --2- 172.21.15.155:0/3337551771 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f10dc0ff630 0x7f10dc0ffa50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f10cc009910 tx=0x7f10cc009c20 comp rx=0 tx=0).stop 2024-03-24T09:41:22.153 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.151+0000 7f10e1d71700 1 -- 172.21.15.155:0/3337551771 shutdown_connections 2024-03-24T09:41:22.153 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.151+0000 7f10e1d71700 1 --2- 172.21.15.155:0/3337551771 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f10dc0ff630 0x7f10dc0ffa50 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:22.153 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.151+0000 7f10e1d71700 1 -- 172.21.15.155:0/3337551771 >> 172.21.15.155:0/3337551771 conn(0x7f10dc0fabd0 msgr2=0x7f10dc0fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:41:22.153 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.151+0000 7f10e1d71700 1 -- 172.21.15.155:0/3337551771 shutdown_connections 2024-03-24T09:41:22.153 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.151+0000 7f10e1d71700 1 -- 172.21.15.155:0/3337551771 wait complete. 2024-03-24T09:41:22.153 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.151+0000 7f10e1d71700 1 Processor -- start 2024-03-24T09:41:22.153 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.151+0000 7f10e1d71700 1 -- start start 2024-03-24T09:41:22.153 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.151+0000 7f10e1d71700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f10dc0ff630 0x7f10dc195c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:41:22.153 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.152+0000 7f10e1d71700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10dc1961a0 con 0x7f10dc0ff630 2024-03-24T09:41:22.153 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.152+0000 7f10db7fe700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f10dc0ff630 0x7f10dc195c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:41:22.153 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.152+0000 7f10db7fe700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f10dc0ff630 0x7f10dc195c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.155:35666/0 (socket says 172.21.15.155:35666) 2024-03-24T09:41:22.153 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.152+0000 7f10db7fe700 1 -- 172.21.15.155:0/1531167728 learned_addr learned my addr 172.21.15.155:0/1531167728 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-24T09:41:22.153 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.152+0000 7f10db7fe700 1 -- 172.21.15.155:0/1531167728 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10cc0095f0 con 0x7f10dc0ff630 2024-03-24T09:41:22.154 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.152+0000 7f10db7fe700 1 --2- 172.21.15.155:0/1531167728 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f10dc0ff630 0x7f10dc195c60 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f10cc004f40 tx=0x7f10cc004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:41:22.154 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.152+0000 7f10d8ff9700 1 -- 172.21.15.155:0/1531167728 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10cc01d070 con 0x7f10dc0ff630 2024-03-24T09:41:22.154 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.152+0000 7f10d8ff9700 1 -- 172.21.15.155:0/1531167728 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f10cc005dc0 con 0x7f10dc0ff630 2024-03-24T09:41:22.154 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.152+0000 7f10d8ff9700 1 -- 172.21.15.155:0/1531167728 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10cc022e40 con 0x7f10dc0ff630 2024-03-24T09:41:22.154 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.152+0000 7f10e1d71700 1 -- 172.21.15.155:0/1531167728 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10dc1963a0 con 0x7f10dc0ff630 2024-03-24T09:41:22.154 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.152+0000 7f10e1d71700 1 -- 172.21.15.155:0/1531167728 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10dc196840 con 0x7f10dc0ff630 2024-03-24T09:41:22.155 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.153+0000 7f10d8ff9700 1 -- 172.21.15.155:0/1531167728 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 13) v1 ==== 45604+0+0 (secure 0 0 0) 0x7f10cc0226c0 con 0x7f10dc0ff630 2024-03-24T09:41:22.155 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.153+0000 7f10e1d71700 1 -- 172.21.15.155:0/1531167728 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10dc0623e0 con 0x7f10dc0ff630 2024-03-24T09:41:22.155 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.153+0000 7f10d8ff9700 1 --2- 172.21.15.155:0/1531167728 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f10c40387c0 0x7f10c403ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:41:22.155 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.154+0000 7f10d8ff9700 1 -- 172.21.15.155:0/1531167728 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f10cc04c840 con 0x7f10dc0ff630 2024-03-24T09:41:22.155 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.154+0000 7f10daffd700 1 --2- 172.21.15.155:0/1531167728 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f10c40387c0 0x7f10c403ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:41:22.155 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.154+0000 7f10daffd700 1 --2- 172.21.15.155:0/1531167728 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f10c40387c0 0x7f10c403ac80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f10d0006fd0 tx=0x7f10d0006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:41:22.159 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.159+0000 7f10d8ff9700 1 -- 172.21.15.155:0/1531167728 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f10cc00b960 con 0x7f10dc0ff630 2024-03-24T09:41:22.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:22 smithi022 ceph-mon[29512]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:41:22.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:22 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:22.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.426+0000 7f10e1d71700 1 -- 172.21.15.155:0/1531167728 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f10dc199130 con 0x7f10dc0ff630 2024-03-24T09:41:22.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.427+0000 7f10d8ff9700 1 -- 172.21.15.155:0/1531167728 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f10cc027070 con 0x7f10dc0ff630 2024-03-24T09:41:22.427 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:41:22.427 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"45356448-e9c2-11ee-95cd-87774f69a715","modified":"2024-03-24T09:39:22.550742Z","created":"2024-03-24T09:39:22.550742Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi022","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:3300","nonce":0},{"type":"v1","addr":"172.21.15.22:6789","nonce":0}]},"addr":"172.21.15.22:6789/0","public_addr":"172.21.15.22:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T09:41:22.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.429+0000 7f10e1d71700 1 -- 172.21.15.155:0/1531167728 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f10c40387c0 msgr2=0x7f10c403ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:22.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.429+0000 7f10e1d71700 1 --2- 172.21.15.155:0/1531167728 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f10c40387c0 0x7f10c403ac80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f10d0006fd0 tx=0x7f10d0006e40 comp rx=0 tx=0).stop 2024-03-24T09:41:22.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.429+0000 7f10e1d71700 1 -- 172.21.15.155:0/1531167728 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f10dc0ff630 msgr2=0x7f10dc195c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:22.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.429+0000 7f10e1d71700 1 --2- 172.21.15.155:0/1531167728 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f10dc0ff630 0x7f10dc195c60 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f10cc004f40 tx=0x7f10cc004740 comp rx=0 tx=0).stop 2024-03-24T09:41:22.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.429+0000 7f10e1d71700 1 -- 172.21.15.155:0/1531167728 shutdown_connections 2024-03-24T09:41:22.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.429+0000 7f10e1d71700 1 --2- 172.21.15.155:0/1531167728 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f10c40387c0 0x7f10c403ac80 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:22.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.429+0000 7f10e1d71700 1 --2- 172.21.15.155:0/1531167728 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f10dc0ff630 0x7f10dc195c60 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:22.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.429+0000 7f10e1d71700 1 -- 172.21.15.155:0/1531167728 >> 172.21.15.155:0/1531167728 conn(0x7f10dc0fabd0 msgr2=0x7f10dc068b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:41:22.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.429+0000 7f10e1d71700 1 -- 172.21.15.155:0/1531167728 shutdown_connections 2024-03-24T09:41:22.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:22.429+0000 7f10e1d71700 1 -- 172.21.15.155:0/1531167728 wait complete. 2024-03-24T09:41:22.430 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-03-24T09:41:23.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:23 smithi022 ceph-mon[29512]: from='client.? 172.21.15.155:0/1531167728' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T09:41:24.001 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T09:41:24.001 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T09:41:24.202 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T09:41:24.248 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T09:41:24.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:24 smithi022 ceph-mon[29512]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:41:25.642 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.641+0000 7ff705ad0700 1 -- 172.21.15.155:0/2538404103 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff700100720 msgr2=0x7ff700100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:25.642 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.641+0000 7ff705ad0700 1 --2- 172.21.15.155:0/2538404103 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff700100720 0x7ff700100b40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff6f4009910 tx=0x7ff6f4009c20 comp rx=0 tx=0).stop 2024-03-24T09:41:25.642 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.641+0000 7ff705ad0700 1 -- 172.21.15.155:0/2538404103 shutdown_connections 2024-03-24T09:41:25.642 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.641+0000 7ff705ad0700 1 --2- 172.21.15.155:0/2538404103 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff700100720 0x7ff700100b40 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:25.643 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.641+0000 7ff705ad0700 1 -- 172.21.15.155:0/2538404103 >> 172.21.15.155:0/2538404103 conn(0x7ff7000fbcc0 msgr2=0x7ff7000fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:41:25.643 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.641+0000 7ff705ad0700 1 -- 172.21.15.155:0/2538404103 shutdown_connections 2024-03-24T09:41:25.643 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.641+0000 7ff705ad0700 1 -- 172.21.15.155:0/2538404103 wait complete. 2024-03-24T09:41:25.643 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.641+0000 7ff705ad0700 1 Processor -- start 2024-03-24T09:41:25.643 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.641+0000 7ff705ad0700 1 -- start start 2024-03-24T09:41:25.643 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.642+0000 7ff705ad0700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff700100720 0x7ff700197db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:41:25.643 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.642+0000 7ff705ad0700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7001982f0 con 0x7ff700100720 2024-03-24T09:41:25.643 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.642+0000 7ff6ff7fe700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff700100720 0x7ff700197db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:41:25.643 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.642+0000 7ff6ff7fe700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff700100720 0x7ff700197db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.155:35684/0 (socket says 172.21.15.155:35684) 2024-03-24T09:41:25.643 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.642+0000 7ff6ff7fe700 1 -- 172.21.15.155:0/3810558288 learned_addr learned my addr 172.21.15.155:0/3810558288 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-24T09:41:25.643 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.642+0000 7ff6ff7fe700 1 -- 172.21.15.155:0/3810558288 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f40095f0 con 0x7ff700100720 2024-03-24T09:41:25.643 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.642+0000 7ff6ff7fe700 1 --2- 172.21.15.155:0/3810558288 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff700100720 0x7ff700197db0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff6f4006010 tx=0x7ff6f4009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:41:25.643 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.642+0000 7ff6fcff9700 1 -- 172.21.15.155:0/3810558288 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6f401d070 con 0x7ff700100720 2024-03-24T09:41:25.644 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.642+0000 7ff705ad0700 1 -- 172.21.15.155:0/3810558288 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7001984f0 con 0x7ff700100720 2024-03-24T09:41:25.644 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.642+0000 7ff6fcff9700 1 -- 172.21.15.155:0/3810558288 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff6f4005dc0 con 0x7ff700100720 2024-03-24T09:41:25.644 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.642+0000 7ff6fcff9700 1 -- 172.21.15.155:0/3810558288 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6f4022e90 con 0x7ff700100720 2024-03-24T09:41:25.644 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.642+0000 7ff705ad0700 1 -- 172.21.15.155:0/3810558288 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff700198990 con 0x7ff700100720 2024-03-24T09:41:25.645 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.643+0000 7ff6fcff9700 1 -- 172.21.15.155:0/3810558288 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 13) v1 ==== 45604+0+0 (secure 0 0 0) 0x7ff6f400b960 con 0x7ff700100720 2024-03-24T09:41:25.645 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.643+0000 7ff705ad0700 1 -- 172.21.15.155:0/3810558288 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff700191950 con 0x7ff700100720 2024-03-24T09:41:25.645 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.643+0000 7ff6fcff9700 1 --2- 172.21.15.155:0/3810558288 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7ff6e8038760 0x7ff6e803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:41:25.645 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.643+0000 7ff6fcff9700 1 -- 172.21.15.155:0/3810558288 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff6f404c840 con 0x7ff700100720 2024-03-24T09:41:25.645 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.644+0000 7ff6feffd700 1 --2- 172.21.15.155:0/3810558288 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7ff6e8038760 0x7ff6e803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:41:25.645 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.644+0000 7ff6feffd700 1 --2- 172.21.15.155:0/3810558288 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7ff6e8038760 0x7ff6e803ac20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0006fd0 tx=0x7ff6f0006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:41:25.648 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.649+0000 7ff6fcff9700 1 -- 172.21.15.155:0/3810558288 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff6f4022750 con 0x7ff700100720 2024-03-24T09:41:25.914 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.914+0000 7ff705ad0700 1 -- 172.21.15.155:0/3810558288 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff7000623e0 con 0x7ff700100720 2024-03-24T09:41:25.914 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.915+0000 7ff6fcff9700 1 -- 172.21.15.155:0/3810558288 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7ff6f4027020 con 0x7ff700100720 2024-03-24T09:41:25.915 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:41:25.915 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"45356448-e9c2-11ee-95cd-87774f69a715","modified":"2024-03-24T09:39:22.550742Z","created":"2024-03-24T09:39:22.550742Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi022","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:3300","nonce":0},{"type":"v1","addr":"172.21.15.22:6789","nonce":0}]},"addr":"172.21.15.22:6789/0","public_addr":"172.21.15.22:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T09:41:25.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.917+0000 7ff705ad0700 1 -- 172.21.15.155:0/3810558288 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7ff6e8038760 msgr2=0x7ff6e803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:25.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.917+0000 7ff705ad0700 1 --2- 172.21.15.155:0/3810558288 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7ff6e8038760 0x7ff6e803ac20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0006fd0 tx=0x7ff6f0006e40 comp rx=0 tx=0).stop 2024-03-24T09:41:25.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.917+0000 7ff705ad0700 1 -- 172.21.15.155:0/3810558288 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff700100720 msgr2=0x7ff700197db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:25.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.917+0000 7ff705ad0700 1 --2- 172.21.15.155:0/3810558288 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff700100720 0x7ff700197db0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff6f4006010 tx=0x7ff6f4009f80 comp rx=0 tx=0).stop 2024-03-24T09:41:25.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.917+0000 7ff705ad0700 1 -- 172.21.15.155:0/3810558288 shutdown_connections 2024-03-24T09:41:25.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.917+0000 7ff705ad0700 1 --2- 172.21.15.155:0/3810558288 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7ff6e8038760 0x7ff6e803ac20 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:25.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.917+0000 7ff705ad0700 1 --2- 172.21.15.155:0/3810558288 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff700100720 0x7ff700197db0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:25.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.917+0000 7ff705ad0700 1 -- 172.21.15.155:0/3810558288 >> 172.21.15.155:0/3810558288 conn(0x7ff7000fbcc0 msgr2=0x7ff7000fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:41:25.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.917+0000 7ff705ad0700 1 -- 172.21.15.155:0/3810558288 shutdown_connections 2024-03-24T09:41:25.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:25.917+0000 7ff705ad0700 1 -- 172.21.15.155:0/3810558288 wait complete. 2024-03-24T09:41:25.918 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-03-24T09:41:26.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:26 smithi022 ceph-mon[29512]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:41:26.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:26 smithi022 ceph-mon[29512]: from='client.? 172.21.15.155:0/3810558288' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T09:41:27.403 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T09:41:27.404 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T09:41:27.608 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T09:41:27.655 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T09:41:28.274 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:28 smithi022 ceph-mon[29512]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:41:29.065 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.064+0000 7f9657a35700 1 -- 172.21.15.155:0/1353901900 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9650100720 msgr2=0x7f9650100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:29.065 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.064+0000 7f9657a35700 1 --2- 172.21.15.155:0/1353901900 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9650100720 0x7f9650100b40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f964c009910 tx=0x7f964c009c20 comp rx=0 tx=0).stop 2024-03-24T09:41:29.065 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.064+0000 7f9657a35700 1 -- 172.21.15.155:0/1353901900 shutdown_connections 2024-03-24T09:41:29.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.064+0000 7f9657a35700 1 --2- 172.21.15.155:0/1353901900 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9650100720 0x7f9650100b40 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:29.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.064+0000 7f9657a35700 1 -- 172.21.15.155:0/1353901900 >> 172.21.15.155:0/1353901900 conn(0x7f96500fbcc0 msgr2=0x7f96500fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:41:29.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.064+0000 7f9657a35700 1 -- 172.21.15.155:0/1353901900 shutdown_connections 2024-03-24T09:41:29.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.064+0000 7f9657a35700 1 -- 172.21.15.155:0/1353901900 wait complete. 2024-03-24T09:41:29.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.065+0000 7f9657a35700 1 Processor -- start 2024-03-24T09:41:29.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.065+0000 7f9657a35700 1 -- start start 2024-03-24T09:41:29.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.065+0000 7f9657a35700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9650100720 0x7f9650074a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:41:29.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.065+0000 7f9657a35700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9650074f60 con 0x7f9650100720 2024-03-24T09:41:29.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.065+0000 7f96557d1700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9650100720 0x7f9650074a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:41:29.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.065+0000 7f96557d1700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9650100720 0x7f9650074a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.155:37784/0 (socket says 172.21.15.155:37784) 2024-03-24T09:41:29.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.065+0000 7f96557d1700 1 -- 172.21.15.155:0/1271183726 learned_addr learned my addr 172.21.15.155:0/1271183726 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-24T09:41:29.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.065+0000 7f96557d1700 1 -- 172.21.15.155:0/1271183726 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f964c0095f0 con 0x7f9650100720 2024-03-24T09:41:29.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.065+0000 7f96557d1700 1 --2- 172.21.15.155:0/1271183726 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9650100720 0x7f9650074a20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f964c006010 tx=0x7f964c009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:41:29.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.066+0000 7f96467fc700 1 -- 172.21.15.155:0/1271183726 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f964c01d070 con 0x7f9650100720 2024-03-24T09:41:29.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.066+0000 7f9657a35700 1 -- 172.21.15.155:0/1271183726 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96500731a0 con 0x7f9650100720 2024-03-24T09:41:29.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.066+0000 7f96467fc700 1 -- 172.21.15.155:0/1271183726 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f964c005dc0 con 0x7f9650100720 2024-03-24T09:41:29.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.066+0000 7f96467fc700 1 -- 172.21.15.155:0/1271183726 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f964c022e90 con 0x7f9650100720 2024-03-24T09:41:29.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.066+0000 7f9657a35700 1 -- 172.21.15.155:0/1271183726 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9650073640 con 0x7f9650100720 2024-03-24T09:41:29.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.067+0000 7f9657a35700 1 -- 172.21.15.155:0/1271183726 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9650106e80 con 0x7f9650100720 2024-03-24T09:41:29.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.067+0000 7f96467fc700 1 -- 172.21.15.155:0/1271183726 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 13) v1 ==== 45604+0+0 (secure 0 0 0) 0x7f964c022470 con 0x7f9650100720 2024-03-24T09:41:29.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.067+0000 7f96467fc700 1 --2- 172.21.15.155:0/1271183726 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f963c038760 0x7f963c03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:41:29.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.067+0000 7f96467fc700 1 -- 172.21.15.155:0/1271183726 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f964c04c7f0 con 0x7f9650100720 2024-03-24T09:41:29.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.067+0000 7f9654fd0700 1 --2- 172.21.15.155:0/1271183726 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f963c038760 0x7f963c03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:41:29.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.068+0000 7f9654fd0700 1 --2- 172.21.15.155:0/1271183726 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f963c038760 0x7f963c03ac20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9640006fd0 tx=0x7f9640006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:41:29.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.072+0000 7f96467fc700 1 -- 172.21.15.155:0/1271183726 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f964c00b960 con 0x7f9650100720 2024-03-24T09:41:29.341 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.341+0000 7f9657a35700 1 -- 172.21.15.155:0/1271183726 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f96500623e0 con 0x7f9650100720 2024-03-24T09:41:29.341 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.341+0000 7f96467fc700 1 -- 172.21.15.155:0/1271183726 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f964c027030 con 0x7f9650100720 2024-03-24T09:41:29.342 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:41:29.342 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"45356448-e9c2-11ee-95cd-87774f69a715","modified":"2024-03-24T09:39:22.550742Z","created":"2024-03-24T09:39:22.550742Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi022","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:3300","nonce":0},{"type":"v1","addr":"172.21.15.22:6789","nonce":0}]},"addr":"172.21.15.22:6789/0","public_addr":"172.21.15.22:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T09:41:29.344 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.344+0000 7f9657a35700 1 -- 172.21.15.155:0/1271183726 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f963c038760 msgr2=0x7f963c03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:29.344 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.344+0000 7f9657a35700 1 --2- 172.21.15.155:0/1271183726 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f963c038760 0x7f963c03ac20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9640006fd0 tx=0x7f9640006e40 comp rx=0 tx=0).stop 2024-03-24T09:41:29.344 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.344+0000 7f9657a35700 1 -- 172.21.15.155:0/1271183726 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9650100720 msgr2=0x7f9650074a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:29.344 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.344+0000 7f9657a35700 1 --2- 172.21.15.155:0/1271183726 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9650100720 0x7f9650074a20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f964c006010 tx=0x7f964c009f80 comp rx=0 tx=0).stop 2024-03-24T09:41:29.344 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.344+0000 7f9657a35700 1 -- 172.21.15.155:0/1271183726 shutdown_connections 2024-03-24T09:41:29.344 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.344+0000 7f9657a35700 1 --2- 172.21.15.155:0/1271183726 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f963c038760 0x7f963c03ac20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:29.344 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.344+0000 7f9657a35700 1 --2- 172.21.15.155:0/1271183726 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9650100720 0x7f9650074a20 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:29.344 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.344+0000 7f9657a35700 1 -- 172.21.15.155:0/1271183726 >> 172.21.15.155:0/1271183726 conn(0x7f96500fbcc0 msgr2=0x7f96500fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:41:29.344 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.344+0000 7f9657a35700 1 -- 172.21.15.155:0/1271183726 shutdown_connections 2024-03-24T09:41:29.344 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:29.344+0000 7f9657a35700 1 -- 172.21.15.155:0/1271183726 wait complete. 2024-03-24T09:41:29.344 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-03-24T09:41:29.389 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:29 smithi022 ceph-mon[29512]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:41:30.323 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:30 smithi022 ceph-mon[29512]: from='client.? 172.21.15.155:0/1271183726' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T09:41:30.925 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T09:41:30.926 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T09:41:31.127 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T09:41:31.172 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T09:41:31.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:31 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:31.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:31 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:31.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:31 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:31.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:31 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-24T09:41:31.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:31 smithi022 ceph-mon[29512]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:41:31.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:31 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:32.653 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.652+0000 7f2d614ba700 1 -- 172.21.15.155:0/2179870692 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2d5c1011c0 msgr2=0x7f2d5c1035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:32.653 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.652+0000 7f2d614ba700 1 --2- 172.21.15.155:0/2179870692 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2d5c1011c0 0x7f2d5c1035b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c009910 tx=0x7f2d4c009c20 comp rx=0 tx=0).stop 2024-03-24T09:41:32.654 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.652+0000 7f2d614ba700 1 -- 172.21.15.155:0/2179870692 shutdown_connections 2024-03-24T09:41:32.654 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.652+0000 7f2d614ba700 1 --2- 172.21.15.155:0/2179870692 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2d5c1011c0 0x7f2d5c1035b0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:32.654 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.652+0000 7f2d614ba700 1 -- 172.21.15.155:0/2179870692 >> 172.21.15.155:0/2179870692 conn(0x7f2d5c0fabd0 msgr2=0x7f2d5c0fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:41:32.654 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.652+0000 7f2d614ba700 1 -- 172.21.15.155:0/2179870692 shutdown_connections 2024-03-24T09:41:32.654 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.652+0000 7f2d614ba700 1 -- 172.21.15.155:0/2179870692 wait complete. 2024-03-24T09:41:32.654 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.652+0000 7f2d614ba700 1 Processor -- start 2024-03-24T09:41:32.654 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.653+0000 7f2d614ba700 1 -- start start 2024-03-24T09:41:32.654 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.653+0000 7f2d614ba700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2d5c1011c0 0x7f2d5c197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:41:32.654 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.653+0000 7f2d614ba700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5c1983a0 con 0x7f2d5c1011c0 2024-03-24T09:41:32.654 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.653+0000 7f2d5affd700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2d5c1011c0 0x7f2d5c197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:41:32.655 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.653+0000 7f2d5affd700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2d5c1011c0 0x7f2d5c197e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.155:37800/0 (socket says 172.21.15.155:37800) 2024-03-24T09:41:32.655 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.653+0000 7f2d5affd700 1 -- 172.21.15.155:0/1076071954 learned_addr learned my addr 172.21.15.155:0/1076071954 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-24T09:41:32.655 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.653+0000 7f2d5affd700 1 -- 172.21.15.155:0/1076071954 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d4c0095f0 con 0x7f2d5c1011c0 2024-03-24T09:41:32.655 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.653+0000 7f2d5affd700 1 --2- 172.21.15.155:0/1076071954 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2d5c1011c0 0x7f2d5c197e60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c005b40 tx=0x7f2d4c005000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:41:32.655 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.654+0000 7f2d58ff9700 1 -- 172.21.15.155:0/1076071954 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d4c01d070 con 0x7f2d5c1011c0 2024-03-24T09:41:32.655 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.654+0000 7f2d58ff9700 1 -- 172.21.15.155:0/1076071954 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2d4c005dc0 con 0x7f2d5c1011c0 2024-03-24T09:41:32.655 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.654+0000 7f2d614ba700 1 -- 172.21.15.155:0/1076071954 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d5c1985a0 con 0x7f2d5c1011c0 2024-03-24T09:41:32.655 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.654+0000 7f2d58ff9700 1 -- 172.21.15.155:0/1076071954 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d4c022e00 con 0x7f2d5c1011c0 2024-03-24T09:41:32.655 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.654+0000 7f2d614ba700 1 -- 172.21.15.155:0/1076071954 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d5c198a40 con 0x7f2d5c1011c0 2024-03-24T09:41:32.656 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.654+0000 7f2d58ff9700 1 -- 172.21.15.155:0/1076071954 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 14) v1 ==== 45618+0+0 (secure 0 0 0) 0x7f2d4c00b960 con 0x7f2d5c1011c0 2024-03-24T09:41:32.656 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.655+0000 7f2d614ba700 1 -- 172.21.15.155:0/1076071954 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d5c191960 con 0x7f2d5c1011c0 2024-03-24T09:41:32.656 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.655+0000 7f2d58ff9700 1 --2- 172.21.15.155:0/1076071954 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f2d480387b0 0x7f2d4803ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:41:32.656 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.655+0000 7f2d58ff9700 1 -- 172.21.15.155:0/1076071954 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2d4c04c920 con 0x7f2d5c1011c0 2024-03-24T09:41:32.656 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.655+0000 7f2d527fc700 1 -- 172.21.15.155:0/1076071954 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f2d480387b0 msgr2=0x7f2d4803ac70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/2979399337 2024-03-24T09:41:32.656 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.655+0000 7f2d527fc700 1 --2- 172.21.15.155:0/1076071954 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f2d480387b0 0x7f2d4803ac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T09:41:32.659 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.660+0000 7f2d58ff9700 1 -- 172.21.15.155:0/1076071954 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2d4c022690 con 0x7f2d5c1011c0 2024-03-24T09:41:32.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:32 smithi022 ceph-mon[29512]: from='mgr.14164 172.21.15.22:0/722756058' entity='mgr.smithi022.emgmxz' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-24T09:41:32.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:32 smithi022 ceph-mon[29512]: mgrmap e14: smithi022.emgmxz(active, since 60s) 2024-03-24T09:41:32.855 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.856+0000 7f2d527fc700 1 -- 172.21.15.155:0/1076071954 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f2d480387b0 msgr2=0x7f2d4803ac70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/2979399337 2024-03-24T09:41:32.856 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.856+0000 7f2d527fc700 1 --2- 172.21.15.155:0/1076071954 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f2d480387b0 0x7f2d4803ac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T09:41:32.929 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.930+0000 7f2d614ba700 1 -- 172.21.15.155:0/1076071954 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2d5c0623e0 con 0x7f2d5c1011c0 2024-03-24T09:41:32.930 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.930+0000 7f2d58ff9700 1 -- 172.21.15.155:0/1076071954 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f2d4c027070 con 0x7f2d5c1011c0 2024-03-24T09:41:32.930 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:41:32.930 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"45356448-e9c2-11ee-95cd-87774f69a715","modified":"2024-03-24T09:39:22.550742Z","created":"2024-03-24T09:39:22.550742Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi022","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:3300","nonce":0},{"type":"v1","addr":"172.21.15.22:6789","nonce":0}]},"addr":"172.21.15.22:6789/0","public_addr":"172.21.15.22:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T09:41:32.932 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.932+0000 7f2d614ba700 1 -- 172.21.15.155:0/1076071954 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f2d480387b0 msgr2=0x7f2d4803ac70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T09:41:32.932 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.932+0000 7f2d614ba700 1 --2- 172.21.15.155:0/1076071954 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f2d480387b0 0x7f2d4803ac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:32.932 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.932+0000 7f2d614ba700 1 -- 172.21.15.155:0/1076071954 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2d5c1011c0 msgr2=0x7f2d5c197e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:32.932 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.932+0000 7f2d614ba700 1 --2- 172.21.15.155:0/1076071954 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2d5c1011c0 0x7f2d5c197e60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c005b40 tx=0x7f2d4c005000 comp rx=0 tx=0).stop 2024-03-24T09:41:32.933 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.932+0000 7f2d614ba700 1 -- 172.21.15.155:0/1076071954 shutdown_connections 2024-03-24T09:41:32.933 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.932+0000 7f2d614ba700 1 --2- 172.21.15.155:0/1076071954 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7f2d480387b0 0x7f2d4803ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:32.933 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.932+0000 7f2d614ba700 1 --2- 172.21.15.155:0/1076071954 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2d5c1011c0 0x7f2d5c197e60 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:32.933 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.932+0000 7f2d614ba700 1 -- 172.21.15.155:0/1076071954 >> 172.21.15.155:0/1076071954 conn(0x7f2d5c0fabd0 msgr2=0x7f2d5c0fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:41:32.933 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.933+0000 7f2d614ba700 1 -- 172.21.15.155:0/1076071954 shutdown_connections 2024-03-24T09:41:32.933 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:32.933+0000 7f2d614ba700 1 -- 172.21.15.155:0/1076071954 wait complete. 2024-03-24T09:41:32.933 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-03-24T09:41:33.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:33 smithi022 ceph-mon[29512]: from='client.? 172.21.15.155:0/1076071954' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T09:41:34.538 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T09:41:34.538 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T09:41:34.744 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T09:41:34.790 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T09:41:36.160 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.157+0000 7fc1ede2c700 1 -- 172.21.15.155:0/284391670 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc1e81011d0 msgr2=0x7fc1e81035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:36.161 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.157+0000 7fc1ede2c700 1 --2- 172.21.15.155:0/284391670 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc1e81011d0 0x7fc1e81035c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc1e000b0a0 tx=0x7fc1e000b3b0 comp rx=0 tx=0).stop 2024-03-24T09:41:36.161 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.157+0000 7fc1ede2c700 1 -- 172.21.15.155:0/284391670 shutdown_connections 2024-03-24T09:41:36.161 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.157+0000 7fc1ede2c700 1 --2- 172.21.15.155:0/284391670 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc1e81011d0 0x7fc1e81035c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:36.161 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.157+0000 7fc1ede2c700 1 -- 172.21.15.155:0/284391670 >> 172.21.15.155:0/284391670 conn(0x7fc1e80fabe0 msgr2=0x7fc1e80fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:41:36.161 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.157+0000 7fc1ede2c700 1 -- 172.21.15.155:0/284391670 shutdown_connections 2024-03-24T09:41:36.161 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.157+0000 7fc1ede2c700 1 -- 172.21.15.155:0/284391670 wait complete. 2024-03-24T09:41:36.161 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.158+0000 7fc1ede2c700 1 Processor -- start 2024-03-24T09:41:36.161 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.158+0000 7fc1ede2c700 1 -- start start 2024-03-24T09:41:36.161 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.158+0000 7fc1ede2c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc1e81011d0 0x7fc1e8195b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:41:36.161 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.158+0000 7fc1ede2c700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e81960c0 con 0x7fc1e81011d0 2024-03-24T09:41:36.161 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.159+0000 7fc1e77fe700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc1e81011d0 0x7fc1e8195b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:41:36.161 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.159+0000 7fc1e77fe700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc1e81011d0 0x7fc1e8195b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.155:37828/0 (socket says 172.21.15.155:37828) 2024-03-24T09:41:36.162 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.159+0000 7fc1e77fe700 1 -- 172.21.15.155:0/3472619316 learned_addr learned my addr 172.21.15.155:0/3472619316 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-24T09:41:36.162 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.159+0000 7fc1e77fe700 1 -- 172.21.15.155:0/3472619316 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e0009d00 con 0x7fc1e81011d0 2024-03-24T09:41:36.162 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.159+0000 7fc1e77fe700 1 --2- 172.21.15.155:0/3472619316 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc1e81011d0 0x7fc1e8195b80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc1e00062a0 tx=0x7fc1e000bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:41:36.162 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.159+0000 7fc1e4ff9700 1 -- 172.21.15.155:0/3472619316 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1e000e040 con 0x7fc1e81011d0 2024-03-24T09:41:36.162 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.159+0000 7fc1e4ff9700 1 -- 172.21.15.155:0/3472619316 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc1e0003d70 con 0x7fc1e81011d0 2024-03-24T09:41:36.162 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.159+0000 7fc1e4ff9700 1 -- 172.21.15.155:0/3472619316 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1e001c8d0 con 0x7fc1e81011d0 2024-03-24T09:41:36.162 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.159+0000 7fc1ede2c700 1 -- 172.21.15.155:0/3472619316 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1e81962c0 con 0x7fc1e81011d0 2024-03-24T09:41:36.162 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.159+0000 7fc1ede2c700 1 -- 172.21.15.155:0/3472619316 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1e8196760 con 0x7fc1e81011d0 2024-03-24T09:41:36.162 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.160+0000 7fc1e4ff9700 1 -- 172.21.15.155:0/3472619316 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 14) v1 ==== 45618+0+0 (secure 0 0 0) 0x7fc1e001cb40 con 0x7fc1e81011d0 2024-03-24T09:41:36.163 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.160+0000 7fc1e4ff9700 1 --2- 172.21.15.155:0/3472619316 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7fc1d4038860 0x7fc1d403ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:41:36.163 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.160+0000 7fc1e4ff9700 1 -- 172.21.15.155:0/3472619316 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc1e004ca90 con 0x7fc1e81011d0 2024-03-24T09:41:36.163 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.160+0000 7fc1e6ffd700 1 -- 172.21.15.155:0/3472619316 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7fc1d4038860 msgr2=0x7fc1d403ad20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/2979399337 2024-03-24T09:41:36.163 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.160+0000 7fc1ede2c700 1 -- 172.21.15.155:0/3472619316 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1e818f570 con 0x7fc1e81011d0 2024-03-24T09:41:36.163 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.160+0000 7fc1e6ffd700 1 --2- 172.21.15.155:0/3472619316 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7fc1d4038860 0x7fc1d403ad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T09:41:36.166 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.165+0000 7fc1e4ff9700 1 -- 172.21.15.155:0/3472619316 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc1e0003f90 con 0x7fc1e81011d0 2024-03-24T09:41:36.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.361+0000 7fc1e6ffd700 1 -- 172.21.15.155:0/3472619316 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7fc1d4038860 msgr2=0x7fc1d403ad20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.22:6800/2979399337 2024-03-24T09:41:36.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.361+0000 7fc1e6ffd700 1 --2- 172.21.15.155:0/3472619316 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7fc1d4038860 0x7fc1d403ad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T09:41:36.436 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.435+0000 7fc1ede2c700 1 -- 172.21.15.155:0/3472619316 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc1e80623e0 con 0x7fc1e81011d0 2024-03-24T09:41:36.436 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.436+0000 7fc1e4ff9700 1 -- 172.21.15.155:0/3472619316 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fc1e001b360 con 0x7fc1e81011d0 2024-03-24T09:41:36.437 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:41:36.437 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"45356448-e9c2-11ee-95cd-87774f69a715","modified":"2024-03-24T09:39:22.550742Z","created":"2024-03-24T09:39:22.550742Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi022","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:3300","nonce":0},{"type":"v1","addr":"172.21.15.22:6789","nonce":0}]},"addr":"172.21.15.22:6789/0","public_addr":"172.21.15.22:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T09:41:36.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.438+0000 7fc1ede2c700 1 -- 172.21.15.155:0/3472619316 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7fc1d4038860 msgr2=0x7fc1d403ad20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T09:41:36.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.438+0000 7fc1ede2c700 1 --2- 172.21.15.155:0/3472619316 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7fc1d4038860 0x7fc1d403ad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:36.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.438+0000 7fc1ede2c700 1 -- 172.21.15.155:0/3472619316 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc1e81011d0 msgr2=0x7fc1e8195b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:36.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.438+0000 7fc1ede2c700 1 --2- 172.21.15.155:0/3472619316 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc1e81011d0 0x7fc1e8195b80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc1e00062a0 tx=0x7fc1e000bd40 comp rx=0 tx=0).stop 2024-03-24T09:41:36.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.438+0000 7fc1ede2c700 1 -- 172.21.15.155:0/3472619316 shutdown_connections 2024-03-24T09:41:36.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.438+0000 7fc1ede2c700 1 --2- 172.21.15.155:0/3472619316 >> [v2:172.21.15.22:6800/2979399337,v1:172.21.15.22:6801/2979399337] conn(0x7fc1d4038860 0x7fc1d403ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:36.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.438+0000 7fc1ede2c700 1 --2- 172.21.15.155:0/3472619316 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc1e81011d0 0x7fc1e8195b80 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:36.440 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.438+0000 7fc1ede2c700 1 -- 172.21.15.155:0/3472619316 >> 172.21.15.155:0/3472619316 conn(0x7fc1e80fabe0 msgr2=0x7fc1e80fb5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:41:36.440 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.438+0000 7fc1ede2c700 1 -- 172.21.15.155:0/3472619316 shutdown_connections 2024-03-24T09:41:36.440 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:36.438+0000 7fc1ede2c700 1 -- 172.21.15.155:0/3472619316 wait complete. 2024-03-24T09:41:36.440 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-03-24T09:41:36.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:36 smithi022 ceph-mon[29512]: from='client.? 172.21.15.155:0/3472619316' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T09:41:37.948 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T09:41:37.948 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T09:41:38.152 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T09:41:38.199 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T09:41:38.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:38 smithi022 ceph-mon[29512]: Active manager daemon smithi022.emgmxz restarted 2024-03-24T09:41:38.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:38 smithi022 ceph-mon[29512]: Activating manager daemon smithi022.emgmxz 2024-03-24T09:41:38.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:38 smithi022 ceph-mon[29512]: osdmap e5: 0 total, 0 up, 0 in 2024-03-24T09:41:38.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:38 smithi022 ceph-mon[29512]: mgrmap e15: smithi022.emgmxz(active, starting, since 0.00306373s) 2024-03-24T09:41:38.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:38 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mon metadata", "id": "smithi022"}]: dispatch 2024-03-24T09:41:38.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:38 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mgr metadata", "who": "smithi022.emgmxz", "id": "smithi022.emgmxz"}]: dispatch 2024-03-24T09:41:38.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:38 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T09:41:38.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:38 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T09:41:38.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:38 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T09:41:38.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:38 smithi022 ceph-mon[29512]: Manager daemon smithi022.emgmxz is now available 2024-03-24T09:41:38.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:38 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:41:38.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:38 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:41:39.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:39.771+0000 7fa47f0c2700 1 -- 172.21.15.155:0/3629095543 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa4780fedd0 msgr2=0x7fa4780ff1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:39.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:39.771+0000 7fa47f0c2700 1 --2- 172.21.15.155:0/3629095543 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa4780fedd0 0x7fa4780ff1f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa46c009910 tx=0x7fa46c009c20 comp rx=0 tx=0).stop 2024-03-24T09:41:39.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:39.771+0000 7fa47f0c2700 1 -- 172.21.15.155:0/3629095543 shutdown_connections 2024-03-24T09:41:39.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:39.771+0000 7fa47f0c2700 1 --2- 172.21.15.155:0/3629095543 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa4780fedd0 0x7fa4780ff1f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:39.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:39.771+0000 7fa47f0c2700 1 -- 172.21.15.155:0/3629095543 >> 172.21.15.155:0/3629095543 conn(0x7fa4780fa370 msgr2=0x7fa4780fc7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:41:39.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:39.771+0000 7fa47f0c2700 1 -- 172.21.15.155:0/3629095543 shutdown_connections 2024-03-24T09:41:39.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:39.771+0000 7fa47f0c2700 1 -- 172.21.15.155:0/3629095543 wait complete. 2024-03-24T09:41:39.774 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:39.772+0000 7fa47f0c2700 1 Processor -- start 2024-03-24T09:41:39.774 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:39.772+0000 7fa47f0c2700 1 -- start start 2024-03-24T09:41:39.774 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:39.772+0000 7fa47f0c2700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa4780fedd0 0x7fa478195c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:41:39.774 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:39.772+0000 7fa47f0c2700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa478196160 con 0x7fa4780fedd0 2024-03-24T09:41:39.775 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:39.772+0000 7fa47ce5e700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa4780fedd0 0x7fa478195c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:41:39.775 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:39.772+0000 7fa47ce5e700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa4780fedd0 0x7fa478195c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.155:42928/0 (socket says 172.21.15.155:42928) 2024-03-24T09:41:39.775 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:39.772+0000 7fa47ce5e700 1 -- 172.21.15.155:0/1002014908 learned_addr learned my addr 172.21.15.155:0/1002014908 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-24T09:41:39.775 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:39.772+0000 7fa47ce5e700 1 -- 172.21.15.155:0/1002014908 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa46c0095f0 con 0x7fa4780fedd0 2024-03-24T09:41:39.775 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:39.773+0000 7fa47ce5e700 1 --2- 172.21.15.155:0/1002014908 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa4780fedd0 0x7fa478195c20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa46c006010 tx=0x7fa46c009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:41:39.775 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:39.773+0000 7fa475ffb700 1 -- 172.21.15.155:0/1002014908 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa46c01d070 con 0x7fa4780fedd0 2024-03-24T09:41:39.775 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:39.773+0000 7fa475ffb700 1 -- 172.21.15.155:0/1002014908 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa46c005dc0 con 0x7fa4780fedd0 2024-03-24T09:41:39.775 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:39.773+0000 7fa47f0c2700 1 -- 172.21.15.155:0/1002014908 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa478196360 con 0x7fa4780fedd0 2024-03-24T09:41:39.775 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:39.773+0000 7fa475ffb700 1 -- 172.21.15.155:0/1002014908 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa46c022e90 con 0x7fa4780fedd0 2024-03-24T09:41:39.775 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:39.773+0000 7fa47f0c2700 1 -- 172.21.15.155:0/1002014908 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa478196800 con 0x7fa4780fedd0 2024-03-24T09:41:39.775 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:39.774+0000 7fa475ffb700 1 -- 172.21.15.155:0/1002014908 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 16) v1 ==== 45469+0+0 (secure 0 0 0) 0x7fa46c00f4b0 con 0x7fa4780fedd0 2024-03-24T09:41:39.776 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:39.774+0000 7fa475ffb700 1 --2- 172.21.15.155:0/1002014908 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fa468038640 0x7fa46803ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:41:39.776 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:39.774+0000 7fa475ffb700 1 -- 172.21.15.155:0/1002014908 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fa46c04d770 con 0x7fa4780fedd0 2024-03-24T09:41:39.776 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:39.774+0000 7fa47f0c2700 1 -- 172.21.15.155:0/1002014908 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa47818fe10 con 0x7fa4780fedd0 2024-03-24T09:41:39.776 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:39.774+0000 7fa477fff700 1 --2- 172.21.15.155:0/1002014908 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fa468038640 0x7fa46803ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:41:39.776 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:39.775+0000 7fa477fff700 1 --2- 172.21.15.155:0/1002014908 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fa468038640 0x7fa46803ab00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa464006fd0 tx=0x7fa464006e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:41:39.780 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:39.779+0000 7fa475ffb700 1 -- 172.21.15.155:0/1002014908 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa46c027070 con 0x7fa4780fedd0 2024-03-24T09:41:39.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:39 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:39.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:39 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:41:39.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:39 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.emgmxz/mirror_snapshot_schedule"}]: dispatch 2024-03-24T09:41:39.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:39 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi022.emgmxz/trash_purge_schedule"}]: dispatch 2024-03-24T09:41:39.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:39 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:39.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:39 smithi022 ceph-mon[29512]: mgrmap e16: smithi022.emgmxz(active, since 1.00621s) 2024-03-24T09:41:40.061 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:40.060+0000 7fa47f0c2700 1 -- 172.21.15.155:0/1002014908 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa4780623e0 con 0x7fa4780fedd0 2024-03-24T09:41:40.061 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:40.060+0000 7fa475ffb700 1 -- 172.21.15.155:0/1002014908 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fa46c017440 con 0x7fa4780fedd0 2024-03-24T09:41:40.062 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:41:40.062 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"45356448-e9c2-11ee-95cd-87774f69a715","modified":"2024-03-24T09:39:22.550742Z","created":"2024-03-24T09:39:22.550742Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi022","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:3300","nonce":0},{"type":"v1","addr":"172.21.15.22:6789","nonce":0}]},"addr":"172.21.15.22:6789/0","public_addr":"172.21.15.22:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T09:41:40.063 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:40.063+0000 7fa47f0c2700 1 -- 172.21.15.155:0/1002014908 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fa468038640 msgr2=0x7fa46803ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:40.064 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:40.063+0000 7fa47f0c2700 1 --2- 172.21.15.155:0/1002014908 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fa468038640 0x7fa46803ab00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa464006fd0 tx=0x7fa464006e40 comp rx=0 tx=0).stop 2024-03-24T09:41:40.064 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:40.063+0000 7fa47f0c2700 1 -- 172.21.15.155:0/1002014908 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa4780fedd0 msgr2=0x7fa478195c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:40.064 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:40.063+0000 7fa47f0c2700 1 --2- 172.21.15.155:0/1002014908 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa4780fedd0 0x7fa478195c20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa46c006010 tx=0x7fa46c009f80 comp rx=0 tx=0).stop 2024-03-24T09:41:40.064 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:40.063+0000 7fa47f0c2700 1 -- 172.21.15.155:0/1002014908 shutdown_connections 2024-03-24T09:41:40.064 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:40.063+0000 7fa47f0c2700 1 --2- 172.21.15.155:0/1002014908 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fa468038640 0x7fa46803ab00 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:40.064 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:40.063+0000 7fa47f0c2700 1 --2- 172.21.15.155:0/1002014908 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa4780fedd0 0x7fa478195c20 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:40.064 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:40.063+0000 7fa47f0c2700 1 -- 172.21.15.155:0/1002014908 >> 172.21.15.155:0/1002014908 conn(0x7fa4780fa370 msgr2=0x7fa4780fac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:41:40.064 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:40.063+0000 7fa47f0c2700 1 -- 172.21.15.155:0/1002014908 shutdown_connections 2024-03-24T09:41:40.064 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:40.063+0000 7fa47f0c2700 1 -- 172.21.15.155:0/1002014908 wait complete. 2024-03-24T09:41:40.065 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-03-24T09:41:41.085 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:40 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:41.086 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:40 smithi022 ceph-mon[29512]: [24/Mar/2024:09:41:40] ENGINE Bus STARTING 2024-03-24T09:41:41.086 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:40 smithi022 ceph-mon[29512]: from='client.? 172.21.15.155:0/1002014908' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T09:41:41.086 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:40 smithi022 ceph-mon[29512]: [24/Mar/2024:09:41:40] ENGINE Serving on https://172.21.15.22:7150 2024-03-24T09:41:41.086 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:40 smithi022 ceph-mon[29512]: [24/Mar/2024:09:41:40] ENGINE Serving on http://172.21.15.22:8765 2024-03-24T09:41:41.086 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:40 smithi022 ceph-mon[29512]: [24/Mar/2024:09:41:40] ENGINE Bus STARTED 2024-03-24T09:41:41.633 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T09:41:41.633 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T09:41:42.406 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:42 smithi022 ceph-mon[29512]: mgrmap e17: smithi022.emgmxz(active, since 2s) 2024-03-24T09:41:42.406 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:42 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:42.816 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T09:41:42.869 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T09:41:44.034 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:43 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:44.034 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:43 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:44.034 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:43 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:44.034 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:43 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:44.034 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:43 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config rm", "who": "osd/host:smithi022", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:41:45.204 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.203+0000 7f743b65e700 1 -- 172.21.15.155:0/2744384625 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f74341066c0 msgr2=0x7f7434106aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:45.204 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.203+0000 7f743b65e700 1 --2- 172.21.15.155:0/2744384625 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f74341066c0 0x7f7434106aa0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f742c009910 tx=0x7f742c009c20 comp rx=0 tx=0).stop 2024-03-24T09:41:45.204 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.203+0000 7f743b65e700 1 -- 172.21.15.155:0/2744384625 shutdown_connections 2024-03-24T09:41:45.205 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.203+0000 7f743b65e700 1 --2- 172.21.15.155:0/2744384625 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f74341066c0 0x7f7434106aa0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:45.205 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.203+0000 7f743b65e700 1 -- 172.21.15.155:0/2744384625 >> 172.21.15.155:0/2744384625 conn(0x7f74340fc1d0 msgr2=0x7f74340fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:41:45.205 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.204+0000 7f743b65e700 1 -- 172.21.15.155:0/2744384625 shutdown_connections 2024-03-24T09:41:45.205 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.204+0000 7f743b65e700 1 -- 172.21.15.155:0/2744384625 wait complete. 2024-03-24T09:41:45.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.204+0000 7f743b65e700 1 Processor -- start 2024-03-24T09:41:45.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.204+0000 7f743b65e700 1 -- start start 2024-03-24T09:41:45.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.204+0000 7f743b65e700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f74341066c0 0x7f7434195f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:41:45.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.204+0000 7f743b65e700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74341964d0 con 0x7f74341066c0 2024-03-24T09:41:45.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.205+0000 7f74393fa700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f74341066c0 0x7f7434195f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:41:45.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.205+0000 7f74393fa700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f74341066c0 0x7f7434195f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.155:42946/0 (socket says 172.21.15.155:42946) 2024-03-24T09:41:45.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.205+0000 7f74393fa700 1 -- 172.21.15.155:0/572494472 learned_addr learned my addr 172.21.15.155:0/572494472 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-24T09:41:45.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.205+0000 7f74393fa700 1 -- 172.21.15.155:0/572494472 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f742c0095f0 con 0x7f74341066c0 2024-03-24T09:41:45.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.205+0000 7f74393fa700 1 --2- 172.21.15.155:0/572494472 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f74341066c0 0x7f7434195f90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f742c0098e0 tx=0x7f742c005520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:41:45.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.205+0000 7f74327fc700 1 -- 172.21.15.155:0/572494472 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f742c01d070 con 0x7f74341066c0 2024-03-24T09:41:45.208 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.205+0000 7f743b65e700 1 -- 172.21.15.155:0/572494472 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7434196730 con 0x7f74341066c0 2024-03-24T09:41:45.208 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.205+0000 7f74327fc700 1 -- 172.21.15.155:0/572494472 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f742c0056b0 con 0x7f74341066c0 2024-03-24T09:41:45.208 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.205+0000 7f74327fc700 1 -- 172.21.15.155:0/572494472 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f742c022d80 con 0x7f74341066c0 2024-03-24T09:41:45.208 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.205+0000 7f743b65e700 1 -- 172.21.15.155:0/572494472 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7434196bb0 con 0x7f74341066c0 2024-03-24T09:41:45.208 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.206+0000 7f74327fc700 1 -- 172.21.15.155:0/572494472 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 17) v1 ==== 45661+0+0 (secure 0 0 0) 0x7f742c00f460 con 0x7f74341066c0 2024-03-24T09:41:45.208 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.206+0000 7f743b65e700 1 -- 172.21.15.155:0/572494472 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7434101e00 con 0x7f74341066c0 2024-03-24T09:41:45.208 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.206+0000 7f74327fc700 1 --2- 172.21.15.155:0/572494472 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f741c038900 0x7f741c03adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:41:45.208 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.206+0000 7f74327fc700 1 -- 172.21.15.155:0/572494472 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f742c04c9f0 con 0x7f74341066c0 2024-03-24T09:41:45.208 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.206+0000 7f7438bf9700 1 --2- 172.21.15.155:0/572494472 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f741c038900 0x7f741c03adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:41:45.208 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.207+0000 7f7438bf9700 1 --2- 172.21.15.155:0/572494472 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f741c038900 0x7f741c03adc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7428006fd0 tx=0x7f7428006e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:41:45.212 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.212+0000 7f74327fc700 1 -- 172.21.15.155:0/572494472 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f742c020bd0 con 0x7f74341066c0 2024-03-24T09:41:45.483 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.482+0000 7f743b65e700 1 -- 172.21.15.155:0/572494472 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f743419a880 con 0x7f74341066c0 2024-03-24T09:41:45.483 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.483+0000 7f74327fc700 1 -- 172.21.15.155:0/572494472 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f742c027030 con 0x7f74341066c0 2024-03-24T09:41:45.484 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:41:45.484 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"45356448-e9c2-11ee-95cd-87774f69a715","modified":"2024-03-24T09:39:22.550742Z","created":"2024-03-24T09:39:22.550742Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi022","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:3300","nonce":0},{"type":"v1","addr":"172.21.15.22:6789","nonce":0}]},"addr":"172.21.15.22:6789/0","public_addr":"172.21.15.22:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T09:41:45.486 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.485+0000 7f743b65e700 1 -- 172.21.15.155:0/572494472 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f741c038900 msgr2=0x7f741c03adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:45.486 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.485+0000 7f743b65e700 1 --2- 172.21.15.155:0/572494472 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f741c038900 0x7f741c03adc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7428006fd0 tx=0x7f7428006e40 comp rx=0 tx=0).stop 2024-03-24T09:41:45.486 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.485+0000 7f743b65e700 1 -- 172.21.15.155:0/572494472 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f74341066c0 msgr2=0x7f7434195f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:45.486 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.485+0000 7f743b65e700 1 --2- 172.21.15.155:0/572494472 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f74341066c0 0x7f7434195f90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f742c0098e0 tx=0x7f742c005520 comp rx=0 tx=0).stop 2024-03-24T09:41:45.486 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.485+0000 7f743b65e700 1 -- 172.21.15.155:0/572494472 shutdown_connections 2024-03-24T09:41:45.486 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.485+0000 7f743b65e700 1 --2- 172.21.15.155:0/572494472 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f741c038900 0x7f741c03adc0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:45.486 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.485+0000 7f743b65e700 1 --2- 172.21.15.155:0/572494472 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f74341066c0 0x7f7434195f90 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:45.486 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.485+0000 7f743b65e700 1 -- 172.21.15.155:0/572494472 >> 172.21.15.155:0/572494472 conn(0x7f74340fc1d0 msgr2=0x7f74340fcf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:41:45.486 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.485+0000 7f743b65e700 1 -- 172.21.15.155:0/572494472 shutdown_connections 2024-03-24T09:41:45.486 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:45.485+0000 7f743b65e700 1 -- 172.21.15.155:0/572494472 wait complete. 2024-03-24T09:41:45.487 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-03-24T09:41:45.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:45 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:45.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:45 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:45.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:45 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:45.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:45 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:45.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:45 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config rm", "who": "osd/host:smithi155", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:41:45.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:45 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:41:45.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:45 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:41:45.533 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:45 smithi022 ceph-mon[29512]: Updating smithi022:/etc/ceph/ceph.conf 2024-03-24T09:41:45.533 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:45 smithi022 ceph-mon[29512]: Updating smithi155:/etc/ceph/ceph.conf 2024-03-24T09:41:45.533 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:45 smithi022 ceph-mon[29512]: Updating smithi155:/var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T09:41:45.533 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:45 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:45.533 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:45 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:46.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:46 smithi022 ceph-mon[29512]: Updating smithi155:/etc/ceph/ceph.client.admin.keyring 2024-03-24T09:41:46.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:46 smithi022 ceph-mon[29512]: Updating smithi155:/var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T09:41:46.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:46 smithi022 ceph-mon[29512]: Updating smithi022:/var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T09:41:46.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:46 smithi022 ceph-mon[29512]: Updating smithi022:/etc/ceph/ceph.client.admin.keyring 2024-03-24T09:41:46.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:46 smithi022 ceph-mon[29512]: from='client.? 172.21.15.155:0/572494472' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T09:41:46.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:46 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:46.533 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:46 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:46.533 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:46 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:46.533 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:46 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi155", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-24T09:41:46.533 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:46 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi155", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-03-24T09:41:46.533 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:46 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:41:47.046 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T09:41:47.046 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T09:41:47.467 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T09:41:47.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:47 smithi022 ceph-mon[29512]: Updating smithi022:/var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T09:41:47.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:47 smithi022 ceph-mon[29512]: Deploying daemon ceph-exporter.smithi155 on smithi155 2024-03-24T09:41:49.344 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.341+0000 7ff2b537a700 1 -- 172.21.15.155:0/600119096 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff2b0101810 msgr2=0x7ff2b0101bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:49.344 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.341+0000 7ff2b537a700 1 --2- 172.21.15.155:0/600119096 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff2b0101810 0x7ff2b0101bf0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0009910 tx=0x7ff2a0009c20 comp rx=0 tx=0).stop 2024-03-24T09:41:49.344 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.341+0000 7ff2b537a700 1 -- 172.21.15.155:0/600119096 shutdown_connections 2024-03-24T09:41:49.344 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.341+0000 7ff2b537a700 1 --2- 172.21.15.155:0/600119096 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff2b0101810 0x7ff2b0101bf0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:49.344 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.341+0000 7ff2b537a700 1 -- 172.21.15.155:0/600119096 >> 172.21.15.155:0/600119096 conn(0x7ff2b00fb110 msgr2=0x7ff2b00fd530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:41:49.345 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.341+0000 7ff2b537a700 1 -- 172.21.15.155:0/600119096 shutdown_connections 2024-03-24T09:41:49.345 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.341+0000 7ff2b537a700 1 -- 172.21.15.155:0/600119096 wait complete. 2024-03-24T09:41:49.345 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.342+0000 7ff2b537a700 1 Processor -- start 2024-03-24T09:41:49.345 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.342+0000 7ff2b537a700 1 -- start start 2024-03-24T09:41:49.345 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.342+0000 7ff2b537a700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff2b0101810 0x7ff2b00ff500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:41:49.345 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.342+0000 7ff2b537a700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b01013c0 con 0x7ff2b0101810 2024-03-24T09:41:49.345 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.343+0000 7ff2aeffd700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff2b0101810 0x7ff2b00ff500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:41:49.345 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.343+0000 7ff2aeffd700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff2b0101810 0x7ff2b00ff500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.155:58182/0 (socket says 172.21.15.155:58182) 2024-03-24T09:41:49.345 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.343+0000 7ff2aeffd700 1 -- 172.21.15.155:0/1286846980 learned_addr learned my addr 172.21.15.155:0/1286846980 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-24T09:41:49.345 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.343+0000 7ff2aeffd700 1 -- 172.21.15.155:0/1286846980 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a00095f0 con 0x7ff2b0101810 2024-03-24T09:41:49.345 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.343+0000 7ff2aeffd700 1 --2- 172.21.15.155:0/1286846980 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff2b0101810 0x7ff2b00ff500 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff2a00098e0 tx=0x7ff2a0005520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:41:49.345 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.343+0000 7ff2acff9700 1 -- 172.21.15.155:0/1286846980 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2a001d070 con 0x7ff2b0101810 2024-03-24T09:41:49.345 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.343+0000 7ff2b537a700 1 -- 172.21.15.155:0/1286846980 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2b00ffa40 con 0x7ff2b0101810 2024-03-24T09:41:49.346 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.343+0000 7ff2acff9700 1 -- 172.21.15.155:0/1286846980 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff2a00056b0 con 0x7ff2b0101810 2024-03-24T09:41:49.346 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.343+0000 7ff2acff9700 1 -- 172.21.15.155:0/1286846980 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2a0022da0 con 0x7ff2b0101810 2024-03-24T09:41:49.346 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.343+0000 7ff2b537a700 1 -- 172.21.15.155:0/1286846980 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2b00ffe60 con 0x7ff2b0101810 2024-03-24T09:41:49.346 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.344+0000 7ff2acff9700 1 -- 172.21.15.155:0/1286846980 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 17) v1 ==== 45661+0+0 (secure 0 0 0) 0x7ff2a000f460 con 0x7ff2b0101810 2024-03-24T09:41:49.347 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.344+0000 7ff2b537a700 1 -- 172.21.15.155:0/1286846980 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2b00693e0 con 0x7ff2b0101810 2024-03-24T09:41:49.347 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.344+0000 7ff2acff9700 1 --2- 172.21.15.155:0/1286846980 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7ff29c0388b0 0x7ff29c03ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:41:49.347 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.345+0000 7ff2acff9700 1 -- 172.21.15.155:0/1286846980 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7ff2a004caf0 con 0x7ff2b0101810 2024-03-24T09:41:49.347 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.345+0000 7ff2a67fc700 1 --2- 172.21.15.155:0/1286846980 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7ff29c0388b0 0x7ff29c03ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:41:49.347 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.345+0000 7ff2a67fc700 1 --2- 172.21.15.155:0/1286846980 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7ff29c0388b0 0x7ff29c03ad70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff298006fd0 tx=0x7ff298006e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:41:49.350 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.350+0000 7ff2acff9700 1 -- 172.21.15.155:0/1286846980 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff2a002c440 con 0x7ff2b0101810 2024-03-24T09:41:49.625 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.625+0000 7ff2b537a700 1 -- 172.21.15.155:0/1286846980 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff2b010ab10 con 0x7ff2b0101810 2024-03-24T09:41:49.626 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.625+0000 7ff2acff9700 1 -- 172.21.15.155:0/1286846980 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7ff2a0027030 con 0x7ff2b0101810 2024-03-24T09:41:49.626 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:41:49.626 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"45356448-e9c2-11ee-95cd-87774f69a715","modified":"2024-03-24T09:39:22.550742Z","created":"2024-03-24T09:39:22.550742Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi022","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:3300","nonce":0},{"type":"v1","addr":"172.21.15.22:6789","nonce":0}]},"addr":"172.21.15.22:6789/0","public_addr":"172.21.15.22:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T09:41:49.629 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.628+0000 7ff2b537a700 1 -- 172.21.15.155:0/1286846980 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7ff29c0388b0 msgr2=0x7ff29c03ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:49.629 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.628+0000 7ff2b537a700 1 --2- 172.21.15.155:0/1286846980 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7ff29c0388b0 0x7ff29c03ad70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff298006fd0 tx=0x7ff298006e40 comp rx=0 tx=0).stop 2024-03-24T09:41:49.629 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.628+0000 7ff2b537a700 1 -- 172.21.15.155:0/1286846980 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff2b0101810 msgr2=0x7ff2b00ff500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:49.629 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.628+0000 7ff2b537a700 1 --2- 172.21.15.155:0/1286846980 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff2b0101810 0x7ff2b00ff500 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff2a00098e0 tx=0x7ff2a0005520 comp rx=0 tx=0).stop 2024-03-24T09:41:49.629 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.628+0000 7ff2b537a700 1 -- 172.21.15.155:0/1286846980 shutdown_connections 2024-03-24T09:41:49.629 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.628+0000 7ff2b537a700 1 --2- 172.21.15.155:0/1286846980 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7ff29c0388b0 0x7ff29c03ad70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:49.629 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.628+0000 7ff2b537a700 1 --2- 172.21.15.155:0/1286846980 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff2b0101810 0x7ff2b00ff500 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:49.629 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.628+0000 7ff2b537a700 1 -- 172.21.15.155:0/1286846980 >> 172.21.15.155:0/1286846980 conn(0x7ff2b00fb110 msgr2=0x7ff2b0102650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:41:49.629 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.628+0000 7ff2b537a700 1 -- 172.21.15.155:0/1286846980 shutdown_connections 2024-03-24T09:41:49.629 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:49.628+0000 7ff2b537a700 1 -- 172.21.15.155:0/1286846980 wait complete. 2024-03-24T09:41:49.630 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-03-24T09:41:51.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:50 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:51.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:50 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:51.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:50 smithi022 ceph-mon[29512]: from='client.? 172.21.15.155:0/1286846980' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T09:41:51.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:50 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:51.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:50 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:51.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:50 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi155", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T09:41:51.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:50 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi155", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-24T09:41:51.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:50 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:41:51.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:50 smithi022 ceph-mon[29512]: Deploying daemon crash.smithi155 on smithi155 2024-03-24T09:41:51.178 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T09:41:51.179 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T09:41:51.394 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T09:41:52.848 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:52.846+0000 7f5477bea700 1 -- 172.21.15.155:0/725770632 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5470101920 msgr2=0x7f5470101d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:52.848 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:52.846+0000 7f5477bea700 1 --2- 172.21.15.155:0/725770632 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5470101920 0x7f5470101d00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f546c009910 tx=0x7f546c009c20 comp rx=0 tx=0).stop 2024-03-24T09:41:52.848 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:52.846+0000 7f5477bea700 1 -- 172.21.15.155:0/725770632 shutdown_connections 2024-03-24T09:41:52.848 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:52.846+0000 7f5477bea700 1 --2- 172.21.15.155:0/725770632 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5470101920 0x7f5470101d00 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:52.848 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:52.846+0000 7f5477bea700 1 -- 172.21.15.155:0/725770632 >> 172.21.15.155:0/725770632 conn(0x7f54700fb110 msgr2=0x7f54700fd530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:41:52.848 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:52.846+0000 7f5477bea700 1 -- 172.21.15.155:0/725770632 shutdown_connections 2024-03-24T09:41:52.848 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:52.846+0000 7f5477bea700 1 -- 172.21.15.155:0/725770632 wait complete. 2024-03-24T09:41:52.848 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:52.847+0000 7f5477bea700 1 Processor -- start 2024-03-24T09:41:52.848 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:52.847+0000 7f5477bea700 1 -- start start 2024-03-24T09:41:52.848 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:52.847+0000 7f5477bea700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5470101920 0x7f54700ff580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:41:52.849 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:52.847+0000 7f5477bea700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5470101440 con 0x7f5470101920 2024-03-24T09:41:52.849 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:52.847+0000 7f5475986700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5470101920 0x7f54700ff580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:41:52.849 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:52.847+0000 7f5475986700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5470101920 0x7f54700ff580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.155:58214/0 (socket says 172.21.15.155:58214) 2024-03-24T09:41:52.849 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:52.847+0000 7f5475986700 1 -- 172.21.15.155:0/1499847590 learned_addr learned my addr 172.21.15.155:0/1499847590 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-24T09:41:52.849 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:52.848+0000 7f5475986700 1 -- 172.21.15.155:0/1499847590 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f546c0095f0 con 0x7f5470101920 2024-03-24T09:41:52.849 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:52.848+0000 7f5475986700 1 --2- 172.21.15.155:0/1499847590 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5470101920 0x7f54700ff580 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f546c005490 tx=0x7f546c005570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:41:52.849 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:52.848+0000 7f5466ffd700 1 -- 172.21.15.155:0/1499847590 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f546c01d070 con 0x7f5470101920 2024-03-24T09:41:52.849 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:52.848+0000 7f5477bea700 1 -- 172.21.15.155:0/1499847590 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54700ffac0 con 0x7f5470101920 2024-03-24T09:41:52.849 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:52.848+0000 7f5466ffd700 1 -- 172.21.15.155:0/1499847590 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f546c005700 con 0x7f5470101920 2024-03-24T09:41:52.849 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:52.848+0000 7f5477bea700 1 -- 172.21.15.155:0/1499847590 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54700ffee0 con 0x7f5470101920 2024-03-24T09:41:52.849 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:52.848+0000 7f5466ffd700 1 -- 172.21.15.155:0/1499847590 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f546c003950 con 0x7f5470101920 2024-03-24T09:41:52.850 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:52.849+0000 7f5477bea700 1 -- 172.21.15.155:0/1499847590 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54540052f0 con 0x7f5470101920 2024-03-24T09:41:52.850 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:52.849+0000 7f5466ffd700 1 -- 172.21.15.155:0/1499847590 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 17) v1 ==== 45661+0+0 (secure 0 0 0) 0x7f546c003c10 con 0x7f5470101920 2024-03-24T09:41:52.850 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:52.849+0000 7f5466ffd700 1 --2- 172.21.15.155:0/1499847590 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f545c0388c0 0x7f545c03ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:41:52.850 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:52.849+0000 7f5466ffd700 1 -- 172.21.15.155:0/1499847590 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f546c04c500 con 0x7f5470101920 2024-03-24T09:41:52.850 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:52.849+0000 7f5475185700 1 --2- 172.21.15.155:0/1499847590 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f545c0388c0 0x7f545c03ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:41:52.850 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:52.850+0000 7f5475185700 1 --2- 172.21.15.155:0/1499847590 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f545c0388c0 0x7f545c03ad80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5460006fd0 tx=0x7f5460006e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:41:52.854 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:52.855+0000 7f5466ffd700 1 -- 172.21.15.155:0/1499847590 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f546c02a590 con 0x7f5470101920 2024-03-24T09:41:53.136 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:53.136+0000 7f5477bea700 1 -- 172.21.15.155:0/1499847590 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5454005160 con 0x7f5470101920 2024-03-24T09:41:53.137 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:53.137+0000 7f5466ffd700 1 -- 172.21.15.155:0/1499847590 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f546c026090 con 0x7f5470101920 2024-03-24T09:41:53.137 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:41:53.137 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"45356448-e9c2-11ee-95cd-87774f69a715","modified":"2024-03-24T09:39:22.550742Z","created":"2024-03-24T09:39:22.550742Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi022","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:3300","nonce":0},{"type":"v1","addr":"172.21.15.22:6789","nonce":0}]},"addr":"172.21.15.22:6789/0","public_addr":"172.21.15.22:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T09:41:53.139 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:53.140+0000 7f5477bea700 1 -- 172.21.15.155:0/1499847590 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f545c0388c0 msgr2=0x7f545c03ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:53.139 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:53.140+0000 7f5477bea700 1 --2- 172.21.15.155:0/1499847590 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f545c0388c0 0x7f545c03ad80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5460006fd0 tx=0x7f5460006e40 comp rx=0 tx=0).stop 2024-03-24T09:41:53.140 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:53.140+0000 7f5477bea700 1 -- 172.21.15.155:0/1499847590 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5470101920 msgr2=0x7f54700ff580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:53.140 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:53.140+0000 7f5477bea700 1 --2- 172.21.15.155:0/1499847590 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5470101920 0x7f54700ff580 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f546c005490 tx=0x7f546c005570 comp rx=0 tx=0).stop 2024-03-24T09:41:53.140 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:53.140+0000 7f5477bea700 1 -- 172.21.15.155:0/1499847590 shutdown_connections 2024-03-24T09:41:53.140 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:53.140+0000 7f5477bea700 1 --2- 172.21.15.155:0/1499847590 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f545c0388c0 0x7f545c03ad80 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:53.140 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:53.140+0000 7f5477bea700 1 --2- 172.21.15.155:0/1499847590 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5470101920 0x7f54700ff580 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:53.140 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:53.140+0000 7f5477bea700 1 -- 172.21.15.155:0/1499847590 >> 172.21.15.155:0/1499847590 conn(0x7f54700fb110 msgr2=0x7f54701026d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:41:53.140 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:53.140+0000 7f5477bea700 1 -- 172.21.15.155:0/1499847590 shutdown_connections 2024-03-24T09:41:53.140 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:53.140+0000 7f5477bea700 1 -- 172.21.15.155:0/1499847590 wait complete. 2024-03-24T09:41:53.140 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-03-24T09:41:53.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:53 smithi022 ceph-mon[29512]: from='client.? 172.21.15.155:0/1499847590' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T09:41:54.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:54 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:54.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:54 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:54.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:54 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:54.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:54 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:54.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:54 smithi022 ceph-mon[29512]: Deploying daemon node-exporter.smithi155 on smithi155 2024-03-24T09:41:54.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:54 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:54.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:54 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:41:54.864 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T09:41:54.864 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T09:41:55.066 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T09:41:57.610 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.568+0000 7f905e7b2700 1 -- 172.21.15.155:0/3980433758 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f90581066c0 msgr2=0x7f9058106aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:57.610 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.568+0000 7f905e7b2700 1 --2- 172.21.15.155:0/3980433758 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f90581066c0 0x7f9058106aa0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9048009910 tx=0x7f9048009c20 comp rx=0 tx=0).stop 2024-03-24T09:41:57.610 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.568+0000 7f905e7b2700 1 -- 172.21.15.155:0/3980433758 shutdown_connections 2024-03-24T09:41:57.610 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.568+0000 7f905e7b2700 1 --2- 172.21.15.155:0/3980433758 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f90581066c0 0x7f9058106aa0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:57.611 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.568+0000 7f905e7b2700 1 -- 172.21.15.155:0/3980433758 >> 172.21.15.155:0/3980433758 conn(0x7f90580fc1d0 msgr2=0x7f90580fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:41:57.611 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.568+0000 7f905e7b2700 1 -- 172.21.15.155:0/3980433758 shutdown_connections 2024-03-24T09:41:57.611 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.568+0000 7f905e7b2700 1 -- 172.21.15.155:0/3980433758 wait complete. 2024-03-24T09:41:57.611 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.569+0000 7f905e7b2700 1 Processor -- start 2024-03-24T09:41:57.611 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.569+0000 7f905e7b2700 1 -- start start 2024-03-24T09:41:57.611 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.569+0000 7f905e7b2700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f90581066c0 0x7f9058198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:41:57.611 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.569+0000 7f905e7b2700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9058198670 con 0x7f90581066c0 2024-03-24T09:41:57.611 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.569+0000 7f9057fff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f90581066c0 0x7f9058198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:41:57.611 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.569+0000 7f9057fff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f90581066c0 0x7f9058198130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.155:34928/0 (socket says 172.21.15.155:34928) 2024-03-24T09:41:57.611 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.569+0000 7f9057fff700 1 -- 172.21.15.155:0/2062207864 learned_addr learned my addr 172.21.15.155:0/2062207864 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-24T09:41:57.611 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.570+0000 7f9057fff700 1 -- 172.21.15.155:0/2062207864 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90480095f0 con 0x7f90581066c0 2024-03-24T09:41:57.611 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.570+0000 7f9057fff700 1 --2- 172.21.15.155:0/2062207864 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f90581066c0 0x7f9058198130 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9048005490 tx=0x7f9048005570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:41:57.611 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.570+0000 7f90557fa700 1 -- 172.21.15.155:0/2062207864 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f904801d070 con 0x7f90581066c0 2024-03-24T09:41:57.612 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.570+0000 7f905e7b2700 1 -- 172.21.15.155:0/2062207864 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90581988d0 con 0x7f90581066c0 2024-03-24T09:41:57.612 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.570+0000 7f90557fa700 1 -- 172.21.15.155:0/2062207864 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9048005700 con 0x7f90581066c0 2024-03-24T09:41:57.612 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.570+0000 7f905e7b2700 1 -- 172.21.15.155:0/2062207864 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9058198d50 con 0x7f90581066c0 2024-03-24T09:41:57.612 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.570+0000 7f90557fa700 1 -- 172.21.15.155:0/2062207864 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90480038d0 con 0x7f90581066c0 2024-03-24T09:41:57.612 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.571+0000 7f905e7b2700 1 -- 172.21.15.155:0/2062207864 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9058101e00 con 0x7f90581066c0 2024-03-24T09:41:57.612 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.571+0000 7f90557fa700 1 -- 172.21.15.155:0/2062207864 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 17) v1 ==== 45661+0+0 (secure 0 0 0) 0x7f904800b6f0 con 0x7f90581066c0 2024-03-24T09:41:57.612 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.571+0000 7f90557fa700 1 --2- 172.21.15.155:0/2062207864 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f90400388c0 0x7f904003ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:41:57.612 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.571+0000 7f90557fa700 1 -- 172.21.15.155:0/2062207864 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f904804c5f0 con 0x7f90581066c0 2024-03-24T09:41:57.612 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.571+0000 7f90577fe700 1 --2- 172.21.15.155:0/2062207864 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f90400388c0 0x7f904003ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:41:57.612 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.572+0000 7f90577fe700 1 --2- 172.21.15.155:0/2062207864 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f90400388c0 0x7f904003ad80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f904c006fd0 tx=0x7f904c006e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:41:57.612 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.576+0000 7f90557fa700 1 -- 172.21.15.155:0/2062207864 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9048004240 con 0x7f90581066c0 2024-03-24T09:41:57.856 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.856+0000 7f905e7b2700 1 -- 172.21.15.155:0/2062207864 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f905819ca20 con 0x7f90581066c0 2024-03-24T09:41:57.857 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.857+0000 7f90557fa700 1 -- 172.21.15.155:0/2062207864 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f9048026030 con 0x7f90581066c0 2024-03-24T09:41:57.857 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:41:57.857 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"45356448-e9c2-11ee-95cd-87774f69a715","modified":"2024-03-24T09:39:22.550742Z","created":"2024-03-24T09:39:22.550742Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi022","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:3300","nonce":0},{"type":"v1","addr":"172.21.15.22:6789","nonce":0}]},"addr":"172.21.15.22:6789/0","public_addr":"172.21.15.22:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T09:41:57.859 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.859+0000 7f905e7b2700 1 -- 172.21.15.155:0/2062207864 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f90400388c0 msgr2=0x7f904003ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:57.860 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.859+0000 7f905e7b2700 1 --2- 172.21.15.155:0/2062207864 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f90400388c0 0x7f904003ad80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f904c006fd0 tx=0x7f904c006e40 comp rx=0 tx=0).stop 2024-03-24T09:41:57.860 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.860+0000 7f905e7b2700 1 -- 172.21.15.155:0/2062207864 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f90581066c0 msgr2=0x7f9058198130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:41:57.860 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.860+0000 7f905e7b2700 1 --2- 172.21.15.155:0/2062207864 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f90581066c0 0x7f9058198130 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9048005490 tx=0x7f9048005570 comp rx=0 tx=0).stop 2024-03-24T09:41:57.860 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.860+0000 7f905e7b2700 1 -- 172.21.15.155:0/2062207864 shutdown_connections 2024-03-24T09:41:57.860 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.860+0000 7f905e7b2700 1 --2- 172.21.15.155:0/2062207864 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f90400388c0 0x7f904003ad80 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:57.860 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.860+0000 7f905e7b2700 1 --2- 172.21.15.155:0/2062207864 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f90581066c0 0x7f9058198130 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:41:57.860 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.860+0000 7f905e7b2700 1 -- 172.21.15.155:0/2062207864 >> 172.21.15.155:0/2062207864 conn(0x7f90580fc1d0 msgr2=0x7f90580fe3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:41:57.860 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.860+0000 7f905e7b2700 1 -- 172.21.15.155:0/2062207864 shutdown_connections 2024-03-24T09:41:57.860 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:41:57.860+0000 7f905e7b2700 1 -- 172.21.15.155:0/2062207864 wait complete. 2024-03-24T09:41:57.860 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-03-24T09:41:58.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:57 smithi022 ceph-mon[29512]: from='client.? 172.21.15.155:0/2062207864' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T09:41:59.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:58 smithi022 ceph-mon[29512]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:41:59.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:58 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:59.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:58 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:59.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:58 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:59.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:58 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:41:59.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:58 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi155.itwzew", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T09:41:59.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:58 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi155.itwzew", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T09:41:59.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:58 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T09:41:59.284 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:58 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:41:59.387 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T09:41:59.387 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T09:42:00.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:41:59 smithi022 ceph-mon[29512]: Deploying daemon mgr.smithi155.itwzew on smithi155 2024-03-24T09:42:00.363 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T09:42:01.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:01 smithi022 ceph-mon[29512]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:02.205 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.204+0000 7f0894b3e700 1 -- 172.21.15.155:0/1791016699 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0890105630 msgr2=0x7f0890105a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:42:02.205 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.204+0000 7f0894b3e700 1 --2- 172.21.15.155:0/1791016699 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0890105630 0x7f0890105a10 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f088800b0a0 tx=0x7f088800b3b0 comp rx=0 tx=0).stop 2024-03-24T09:42:02.205 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.204+0000 7f0894b3e700 1 -- 172.21.15.155:0/1791016699 shutdown_connections 2024-03-24T09:42:02.206 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.204+0000 7f0894b3e700 1 --2- 172.21.15.155:0/1791016699 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0890105630 0x7f0890105a10 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:42:02.206 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.204+0000 7f0894b3e700 1 -- 172.21.15.155:0/1791016699 >> 172.21.15.155:0/1791016699 conn(0x7f0890075110 msgr2=0x7f0890075520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:42:02.206 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.204+0000 7f0894b3e700 1 -- 172.21.15.155:0/1791016699 shutdown_connections 2024-03-24T09:42:02.206 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.204+0000 7f0894b3e700 1 -- 172.21.15.155:0/1791016699 wait complete. 2024-03-24T09:42:02.206 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.204+0000 7f0894b3e700 1 Processor -- start 2024-03-24T09:42:02.206 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.205+0000 7f0894b3e700 1 -- start start 2024-03-24T09:42:02.206 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.205+0000 7f0894b3e700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0890105630 0x7f0890195f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:42:02.206 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.205+0000 7f0894b3e700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0890196460 con 0x7f0890105630 2024-03-24T09:42:02.206 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.205+0000 7f088e59c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0890105630 0x7f0890195f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:42:02.206 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.205+0000 7f088e59c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0890105630 0x7f0890195f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.155:34948/0 (socket says 172.21.15.155:34948) 2024-03-24T09:42:02.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.205+0000 7f088e59c700 1 -- 172.21.15.155:0/3517071160 learned_addr learned my addr 172.21.15.155:0/3517071160 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-24T09:42:02.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.205+0000 7f088e59c700 1 -- 172.21.15.155:0/3517071160 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0888009d00 con 0x7f0890105630 2024-03-24T09:42:02.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.205+0000 7f088e59c700 1 --2- 172.21.15.155:0/3517071160 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0890105630 0x7f0890195f20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f08880062a0 tx=0x7f088800bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:42:02.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.206+0000 7f08877fe700 1 -- 172.21.15.155:0/3517071160 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f088800e040 con 0x7f0890105630 2024-03-24T09:42:02.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.206+0000 7f08877fe700 1 -- 172.21.15.155:0/3517071160 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0888007a40 con 0x7f0890105630 2024-03-24T09:42:02.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.206+0000 7f0894b3e700 1 -- 172.21.15.155:0/3517071160 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0890196660 con 0x7f0890105630 2024-03-24T09:42:02.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.206+0000 7f08877fe700 1 -- 172.21.15.155:0/3517071160 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0888012430 con 0x7f0890105630 2024-03-24T09:42:02.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.206+0000 7f0894b3e700 1 -- 172.21.15.155:0/3517071160 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0890196b00 con 0x7f0890105630 2024-03-24T09:42:02.208 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.206+0000 7f0894b3e700 1 -- 172.21.15.155:0/3517071160 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0890100d00 con 0x7f0890105630 2024-03-24T09:42:02.208 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.207+0000 7f08877fe700 1 -- 172.21.15.155:0/3517071160 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 17) v1 ==== 45661+0+0 (secure 0 0 0) 0x7f0888004590 con 0x7f0890105630 2024-03-24T09:42:02.208 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.207+0000 7f08877fe700 1 --2- 172.21.15.155:0/3517071160 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f087c0388b0 0x7f087c03ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:42:02.208 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.207+0000 7f08877fe700 1 -- 172.21.15.155:0/3517071160 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f088804c6b0 con 0x7f0890105630 2024-03-24T09:42:02.208 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.207+0000 7f088dd9b700 1 --2- 172.21.15.155:0/3517071160 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f087c0388b0 0x7f087c03ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:42:02.208 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.207+0000 7f088dd9b700 1 --2- 172.21.15.155:0/3517071160 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f087c0388b0 0x7f087c03ad70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0878006fd0 tx=0x7f0878006e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:42:02.212 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.212+0000 7f08877fe700 1 -- 172.21.15.155:0/3517071160 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f088801aa00 con 0x7f0890105630 2024-03-24T09:42:02.498 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.498+0000 7f0894b3e700 1 -- 172.21.15.155:0/3517071160 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f089019a810 con 0x7f0890105630 2024-03-24T09:42:02.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.499+0000 7f08877fe700 1 -- 172.21.15.155:0/3517071160 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f088804bed0 con 0x7f0890105630 2024-03-24T09:42:02.499 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:42:02.499 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"45356448-e9c2-11ee-95cd-87774f69a715","modified":"2024-03-24T09:39:22.550742Z","created":"2024-03-24T09:39:22.550742Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi022","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:3300","nonce":0},{"type":"v1","addr":"172.21.15.22:6789","nonce":0}]},"addr":"172.21.15.22:6789/0","public_addr":"172.21.15.22:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T09:42:02.501 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.501+0000 7f0894b3e700 1 -- 172.21.15.155:0/3517071160 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f087c0388b0 msgr2=0x7f087c03ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:42:02.501 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.501+0000 7f0894b3e700 1 --2- 172.21.15.155:0/3517071160 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f087c0388b0 0x7f087c03ad70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0878006fd0 tx=0x7f0878006e40 comp rx=0 tx=0).stop 2024-03-24T09:42:02.501 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.501+0000 7f0894b3e700 1 -- 172.21.15.155:0/3517071160 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0890105630 msgr2=0x7f0890195f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:42:02.501 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.501+0000 7f0894b3e700 1 --2- 172.21.15.155:0/3517071160 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0890105630 0x7f0890195f20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f08880062a0 tx=0x7f088800bd40 comp rx=0 tx=0).stop 2024-03-24T09:42:02.501 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.501+0000 7f0894b3e700 1 -- 172.21.15.155:0/3517071160 shutdown_connections 2024-03-24T09:42:02.501 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.501+0000 7f0894b3e700 1 --2- 172.21.15.155:0/3517071160 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f087c0388b0 0x7f087c03ad70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:42:02.502 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.501+0000 7f0894b3e700 1 --2- 172.21.15.155:0/3517071160 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0890105630 0x7f0890195f20 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:42:02.502 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.501+0000 7f0894b3e700 1 -- 172.21.15.155:0/3517071160 >> 172.21.15.155:0/3517071160 conn(0x7f0890075110 msgr2=0x7f08901031a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:42:02.502 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.501+0000 7f0894b3e700 1 -- 172.21.15.155:0/3517071160 shutdown_connections 2024-03-24T09:42:02.502 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:02.501+0000 7f0894b3e700 1 -- 172.21.15.155:0/3517071160 wait complete. 2024-03-24T09:42:02.502 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-03-24T09:42:03.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:03 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:03.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:03 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:03.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:03 smithi022 ceph-mon[29512]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:03.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:03 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:03.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:03 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:03.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:03 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T09:42:03.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:03 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:42:03.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:03 smithi022 ceph-mon[29512]: Deploying daemon mon.smithi155 on smithi155 2024-03-24T09:42:03.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:03 smithi022 ceph-mon[29512]: from='client.? 172.21.15.155:0/3517071160' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T09:42:04.095 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T09:42:04.096 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T09:42:04.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:04 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:04.848 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T09:42:05.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:05 smithi022 ceph-mon[29512]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:06.996 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:06.995+0000 7fb7bed88700 1 -- 172.21.15.155:0/707903282 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb7b81066b0 msgr2=0x7fb7b8106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:42:06.997 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:06.995+0000 7fb7bed88700 1 --2- 172.21.15.155:0/707903282 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb7b81066b0 0x7fb7b8106a90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac009910 tx=0x7fb7ac009c20 comp rx=0 tx=0).stop 2024-03-24T09:42:06.997 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:06.995+0000 7fb7bed88700 1 -- 172.21.15.155:0/707903282 shutdown_connections 2024-03-24T09:42:06.997 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:06.995+0000 7fb7bed88700 1 --2- 172.21.15.155:0/707903282 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb7b81066b0 0x7fb7b8106a90 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:42:06.997 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:06.995+0000 7fb7bed88700 1 -- 172.21.15.155:0/707903282 >> 172.21.15.155:0/707903282 conn(0x7fb7b80fa820 msgr2=0x7fb7b80fcc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:42:06.997 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:06.995+0000 7fb7bed88700 1 -- 172.21.15.155:0/707903282 shutdown_connections 2024-03-24T09:42:06.997 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:06.995+0000 7fb7bed88700 1 -- 172.21.15.155:0/707903282 wait complete. 2024-03-24T09:42:06.997 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:06.996+0000 7fb7bed88700 1 Processor -- start 2024-03-24T09:42:06.997 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:06.996+0000 7fb7bed88700 1 -- start start 2024-03-24T09:42:06.997 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:06.996+0000 7fb7bed88700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb7b81066b0 0x7fb7b80ffda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:42:06.997 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:06.996+0000 7fb7bed88700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7b8101c60 con 0x7fb7b81066b0 2024-03-24T09:42:06.997 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:06.996+0000 7fb7bcb24700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb7b81066b0 0x7fb7b80ffda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:42:06.998 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:06.996+0000 7fb7bcb24700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb7b81066b0 0x7fb7b80ffda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.155:34980/0 (socket says 172.21.15.155:34980) 2024-03-24T09:42:06.998 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:06.996+0000 7fb7bcb24700 1 -- 172.21.15.155:0/3699955722 learned_addr learned my addr 172.21.15.155:0/3699955722 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-24T09:42:06.998 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:06.996+0000 7fb7bcb24700 1 -- 172.21.15.155:0/3699955722 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7ac0095f0 con 0x7fb7b81066b0 2024-03-24T09:42:06.998 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:06.997+0000 7fb7bcb24700 1 --2- 172.21.15.155:0/3699955722 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb7b81066b0 0x7fb7b80ffda0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac005490 tx=0x7fb7ac005570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:42:06.998 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:06.997+0000 7fb7b5ffb700 1 -- 172.21.15.155:0/3699955722 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7ac01d070 con 0x7fb7b81066b0 2024-03-24T09:42:06.998 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:06.997+0000 7fb7b5ffb700 1 -- 172.21.15.155:0/3699955722 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7ac005700 con 0x7fb7b81066b0 2024-03-24T09:42:06.998 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:06.997+0000 7fb7bed88700 1 -- 172.21.15.155:0/3699955722 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7b81002e0 con 0x7fb7b81066b0 2024-03-24T09:42:06.998 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:06.997+0000 7fb7b5ffb700 1 -- 172.21.15.155:0/3699955722 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7ac0038e0 con 0x7fb7b81066b0 2024-03-24T09:42:06.998 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:06.997+0000 7fb7bed88700 1 -- 172.21.15.155:0/3699955722 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7b8100760 con 0x7fb7b81066b0 2024-03-24T09:42:06.999 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:06.998+0000 7fb7bed88700 1 -- 172.21.15.155:0/3699955722 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7b80623e0 con 0x7fb7b81066b0 2024-03-24T09:42:06.999 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:06.998+0000 7fb7b5ffb700 1 -- 172.21.15.155:0/3699955722 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 17) v1 ==== 45661+0+0 (secure 0 0 0) 0x7fb7ac003ba0 con 0x7fb7b81066b0 2024-03-24T09:42:06.999 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:06.998+0000 7fb7b5ffb700 1 --2- 172.21.15.155:0/3699955722 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fb7a80388c0 0x7fb7a803ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:42:06.999 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:06.998+0000 7fb7b5ffb700 1 -- 172.21.15.155:0/3699955722 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fb7ac04c6f0 con 0x7fb7b81066b0 2024-03-24T09:42:06.999 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:06.998+0000 7fb7b7fff700 1 --2- 172.21.15.155:0/3699955722 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fb7a80388c0 0x7fb7a803ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:42:06.999 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:06.999+0000 7fb7b7fff700 1 --2- 172.21.15.155:0/3699955722 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fb7a80388c0 0x7fb7a803ad80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4006fd0 tx=0x7fb7a4006e40 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:42:07.003 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:07.003+0000 7fb7b5ffb700 1 -- 172.21.15.155:0/3699955722 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb7ac02aa00 con 0x7fb7b81066b0 2024-03-24T09:42:07.294 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:07.293+0000 7fb7bed88700 1 -- 172.21.15.155:0/3699955722 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb7b81009a0 con 0x7fb7b81066b0 2024-03-24T09:42:07.294 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:07.294+0000 7fb7b5ffb700 1 -- 172.21.15.155:0/3699955722 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fb7ac026070 con 0x7fb7b81066b0 2024-03-24T09:42:07.294 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:42:07.294 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"45356448-e9c2-11ee-95cd-87774f69a715","modified":"2024-03-24T09:39:22.550742Z","created":"2024-03-24T09:39:22.550742Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi022","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:3300","nonce":0},{"type":"v1","addr":"172.21.15.22:6789","nonce":0}]},"addr":"172.21.15.22:6789/0","public_addr":"172.21.15.22:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T09:42:07.297 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:07.297+0000 7fb7bed88700 1 -- 172.21.15.155:0/3699955722 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fb7a80388c0 msgr2=0x7fb7a803ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:42:07.297 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:07.297+0000 7fb7bed88700 1 --2- 172.21.15.155:0/3699955722 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fb7a80388c0 0x7fb7a803ad80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4006fd0 tx=0x7fb7a4006e40 comp rx=0 tx=0).stop 2024-03-24T09:42:07.297 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:07.297+0000 7fb7bed88700 1 -- 172.21.15.155:0/3699955722 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb7b81066b0 msgr2=0x7fb7b80ffda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:42:07.297 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:07.297+0000 7fb7bed88700 1 --2- 172.21.15.155:0/3699955722 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb7b81066b0 0x7fb7b80ffda0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac005490 tx=0x7fb7ac005570 comp rx=0 tx=0).stop 2024-03-24T09:42:07.297 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:07.297+0000 7fb7bed88700 1 -- 172.21.15.155:0/3699955722 shutdown_connections 2024-03-24T09:42:07.297 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:07.297+0000 7fb7bed88700 1 --2- 172.21.15.155:0/3699955722 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fb7a80388c0 0x7fb7a803ad80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:42:07.297 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:07.297+0000 7fb7bed88700 1 --2- 172.21.15.155:0/3699955722 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb7b81066b0 0x7fb7b80ffda0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:42:07.297 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:07.297+0000 7fb7bed88700 1 -- 172.21.15.155:0/3699955722 >> 172.21.15.155:0/3699955722 conn(0x7fb7b80fa820 msgr2=0x7fb7b80fac00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:42:07.297 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:07.297+0000 7fb7bed88700 1 -- 172.21.15.155:0/3699955722 shutdown_connections 2024-03-24T09:42:07.297 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:07.297+0000 7fb7bed88700 1 -- 172.21.15.155:0/3699955722 wait complete. 2024-03-24T09:42:07.299 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-03-24T09:42:07.502 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:07 smithi155 ceph-mon[37016]: mon.smithi155@-1(synchronizing).paxosservice(auth 1..8) refresh upgraded, format 0 -> 3 2024-03-24T09:42:07.797 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:07 smithi155 bash[36954]: 9188cfee04216974bad6d5ac4a77dd9aaf6bfea5580ea433f3207e6c8c09b902 2024-03-24T09:42:07.797 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:07 smithi155 systemd[1]: Started Ceph mon.smithi155 for 45356448-e9c2-11ee-95cd-87774f69a715. 2024-03-24T09:42:09.095 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T09:42:09.095 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T09:42:09.307 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi155/config 2024-03-24T09:42:12.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:12 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mon metadata", "id": "smithi022"}]: dispatch 2024-03-24T09:42:12.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:12 smithi155 ceph-mon[37016]: mon.smithi022 calling monitor election 2024-03-24T09:42:12.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:12 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mon metadata", "id": "smithi155"}]: dispatch 2024-03-24T09:42:12.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:12 smithi155 ceph-mon[37016]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:12.963 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:12 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mon metadata", "id": "smithi155"}]: dispatch 2024-03-24T09:42:12.963 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:12 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:42:12.963 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:12 smithi155 ceph-mon[37016]: from='mgr.? 172.21.15.155:0/2469825073' entity='mgr.smithi155.itwzew' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi155.itwzew/crt"}]: dispatch 2024-03-24T09:42:12.963 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:12 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mon metadata", "id": "smithi155"}]: dispatch 2024-03-24T09:42:12.963 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:12 smithi155 ceph-mon[37016]: mon.smithi155 calling monitor election 2024-03-24T09:42:12.963 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:12 smithi155 ceph-mon[37016]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:12.963 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:12 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mon metadata", "id": "smithi155"}]: dispatch 2024-03-24T09:42:12.963 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:12 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mon metadata", "id": "smithi155"}]: dispatch 2024-03-24T09:42:12.963 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:12 smithi155 ceph-mon[37016]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:12.963 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:12 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mon metadata", "id": "smithi155"}]: dispatch 2024-03-24T09:42:12.963 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:12 smithi155 ceph-mon[37016]: mon.smithi022 is new leader, mons smithi022,smithi155 in quorum (ranks 0,1) 2024-03-24T09:42:12.963 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:12 smithi155 ceph-mon[37016]: monmap e2: 2 mons at {smithi022=[v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0],smithi155=[v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T09:42:12.963 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:12 smithi155 ceph-mon[37016]: fsmap 2024-03-24T09:42:12.964 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:12 smithi155 ceph-mon[37016]: osdmap e5: 0 total, 0 up, 0 in 2024-03-24T09:42:12.964 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:12 smithi155 ceph-mon[37016]: mgrmap e17: smithi022.emgmxz(active, since 34s) 2024-03-24T09:42:12.964 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:12 smithi155 ceph-mon[37016]: Standby manager daemon smithi155.itwzew started 2024-03-24T09:42:12.964 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:12 smithi155 ceph-mon[37016]: from='mgr.? 172.21.15.155:0/2469825073' entity='mgr.smithi155.itwzew' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T09:42:12.964 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:12 smithi155 ceph-mon[37016]: overall HEALTH_OK 2024-03-24T09:42:12.964 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:12 smithi155 ceph-mon[37016]: from='mgr.? 172.21.15.155:0/2469825073' entity='mgr.smithi155.itwzew' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi155.itwzew/key"}]: dispatch 2024-03-24T09:42:12.964 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:12 smithi155 ceph-mon[37016]: from='mgr.? 172.21.15.155:0/2469825073' entity='mgr.smithi155.itwzew' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T09:42:12.964 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:12 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:12.964 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:12 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:12.964 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:12 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:12.964 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:12 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:12.964 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:12 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:13.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:12 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mon metadata", "id": "smithi022"}]: dispatch 2024-03-24T09:42:13.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:12 smithi022 ceph-mon[29512]: mon.smithi022 calling monitor election 2024-03-24T09:42:13.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:12 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mon metadata", "id": "smithi155"}]: dispatch 2024-03-24T09:42:13.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:12 smithi022 ceph-mon[29512]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:13.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:12 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mon metadata", "id": "smithi155"}]: dispatch 2024-03-24T09:42:13.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:12 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:42:13.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:12 smithi022 ceph-mon[29512]: from='mgr.? 172.21.15.155:0/2469825073' entity='mgr.smithi155.itwzew' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi155.itwzew/crt"}]: dispatch 2024-03-24T09:42:13.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:12 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mon metadata", "id": "smithi155"}]: dispatch 2024-03-24T09:42:13.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:12 smithi022 ceph-mon[29512]: mon.smithi155 calling monitor election 2024-03-24T09:42:13.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:12 smithi022 ceph-mon[29512]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:13.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:12 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mon metadata", "id": "smithi155"}]: dispatch 2024-03-24T09:42:13.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:12 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mon metadata", "id": "smithi155"}]: dispatch 2024-03-24T09:42:13.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:12 smithi022 ceph-mon[29512]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:13.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:12 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mon metadata", "id": "smithi155"}]: dispatch 2024-03-24T09:42:13.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:12 smithi022 ceph-mon[29512]: mon.smithi022 is new leader, mons smithi022,smithi155 in quorum (ranks 0,1) 2024-03-24T09:42:13.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:12 smithi022 ceph-mon[29512]: monmap e2: 2 mons at {smithi022=[v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0],smithi155=[v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T09:42:13.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:12 smithi022 ceph-mon[29512]: fsmap 2024-03-24T09:42:13.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:12 smithi022 ceph-mon[29512]: osdmap e5: 0 total, 0 up, 0 in 2024-03-24T09:42:13.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:12 smithi022 ceph-mon[29512]: mgrmap e17: smithi022.emgmxz(active, since 34s) 2024-03-24T09:42:13.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:12 smithi022 ceph-mon[29512]: Standby manager daemon smithi155.itwzew started 2024-03-24T09:42:13.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:12 smithi022 ceph-mon[29512]: from='mgr.? 172.21.15.155:0/2469825073' entity='mgr.smithi155.itwzew' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T09:42:13.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:12 smithi022 ceph-mon[29512]: overall HEALTH_OK 2024-03-24T09:42:13.034 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:12 smithi022 ceph-mon[29512]: from='mgr.? 172.21.15.155:0/2469825073' entity='mgr.smithi155.itwzew' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi155.itwzew/key"}]: dispatch 2024-03-24T09:42:13.034 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:12 smithi022 ceph-mon[29512]: from='mgr.? 172.21.15.155:0/2469825073' entity='mgr.smithi155.itwzew' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T09:42:13.034 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:12 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:13.034 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:12 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:13.034 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:12 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:13.034 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:12 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:13.034 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:12 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:13.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:13 smithi155 ceph-mon[37016]: mgrmap e18: smithi022.emgmxz(active, since 34s), standbys: smithi155.itwzew 2024-03-24T09:42:13.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:13 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mgr metadata", "who": "smithi155.itwzew", "id": "smithi155.itwzew"}]: dispatch 2024-03-24T09:42:13.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:13 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:42:13.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:13 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mon metadata", "id": "smithi155"}]: dispatch 2024-03-24T09:42:13.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:13 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:13.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:13.967+0000 7f38343df700 1 -- 172.21.15.155:0/3951914269 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f38180057d0 msgr2=0x7f3818007c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:42:13.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:13.967+0000 7f38343df700 1 --2- 172.21.15.155:0/3951914269 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f38180057d0 0x7f3818007c60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f382c0ff620 tx=0x7f381c00feb0 comp rx=0 tx=0).stop 2024-03-24T09:42:13.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:13.967+0000 7f38343df700 1 -- 172.21.15.155:0/3951914269 shutdown_connections 2024-03-24T09:42:13.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:13.967+0000 7f38343df700 1 --2- 172.21.15.155:0/3951914269 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f38180057d0 0x7f3818007c60 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:42:13.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:13.967+0000 7f38343df700 1 -- 172.21.15.155:0/3951914269 >> 172.21.15.155:0/3951914269 conn(0x7f382c076390 msgr2=0x7f382c0fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:42:13.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:13.967+0000 7f38343df700 1 -- 172.21.15.155:0/3951914269 shutdown_connections 2024-03-24T09:42:13.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:13.967+0000 7f38343df700 1 -- 172.21.15.155:0/3951914269 wait complete. 2024-03-24T09:42:13.969 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:13.968+0000 7f38343df700 1 Processor -- start 2024-03-24T09:42:13.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:13.968+0000 7f38343df700 1 -- start start 2024-03-24T09:42:13.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:13.968+0000 7f38343df700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38180057d0 0x7f382c193f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:42:13.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:13.968+0000 7f38343df700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f382c194450 0x7f382c198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:42:13.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:13.968+0000 7f38343df700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382c194a70 con 0x7f382c194450 2024-03-24T09:42:13.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:13.968+0000 7f38343df700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382c194be0 con 0x7f38180057d0 2024-03-24T09:42:13.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:13.968+0000 7f383217b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38180057d0 0x7f382c193f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:42:13.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:13.968+0000 7f383217b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38180057d0 0x7f382c193f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:42084/0 (socket says 172.21.15.155:42084) 2024-03-24T09:42:13.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:13.968+0000 7f383217b700 1 -- 172.21.15.155:0/1009428200 learned_addr learned my addr 172.21.15.155:0/1009428200 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-24T09:42:13.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:13.968+0000 7f383197a700 1 --2- 172.21.15.155:0/1009428200 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f382c194450 0x7f382c198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:42:13.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:13.968+0000 7f383217b700 1 -- 172.21.15.155:0/1009428200 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38180057d0 msgr2=0x7f382c193f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-24T09:42:13.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:13.968+0000 7f383217b700 1 -- 172.21.15.155:0/1009428200 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38180057d0 msgr2=0x7f382c193f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-24T09:42:13.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:13.968+0000 7f383217b700 1 --2- 172.21.15.155:0/1009428200 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38180057d0 0x7f382c193f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T09:42:13.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:13.968+0000 7f383217b700 1 --2- 172.21.15.155:0/1009428200 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38180057d0 0x7f382c193f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T09:42:13.971 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:13.969+0000 7f383197a700 1 -- 172.21.15.155:0/1009428200 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38180057d0 msgr2=0x7f382c193f10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T09:42:13.971 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:13.969+0000 7f383197a700 1 --2- 172.21.15.155:0/1009428200 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38180057d0 0x7f382c193f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:42:13.971 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:13.969+0000 7f383197a700 1 -- 172.21.15.155:0/1009428200 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f381c00f900 con 0x7f382c194450 2024-03-24T09:42:13.971 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:13.969+0000 7f383197a700 1 --2- 172.21.15.155:0/1009428200 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f382c194450 0x7f382c198800 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f382800b4d0 tx=0x7f382800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:42:13.971 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:13.969+0000 7f38237fe700 1 -- 172.21.15.155:0/1009428200 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3828011800 con 0x7f382c194450 2024-03-24T09:42:13.971 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:13.969+0000 7f38237fe700 1 -- 172.21.15.155:0/1009428200 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3828011e20 con 0x7f382c194450 2024-03-24T09:42:13.971 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:13.969+0000 7f38343df700 1 -- 172.21.15.155:0/1009428200 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f382c198e00 con 0x7f382c194450 2024-03-24T09:42:13.971 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:13.969+0000 7f38237fe700 1 -- 172.21.15.155:0/1009428200 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f382800f2e0 con 0x7f382c194450 2024-03-24T09:42:13.971 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:13.969+0000 7f38343df700 1 -- 172.21.15.155:0/1009428200 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f382c199350 con 0x7f382c194450 2024-03-24T09:42:13.971 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:13.970+0000 7f38343df700 1 -- 172.21.15.155:0/1009428200 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f382c18dea0 con 0x7f382c194450 2024-03-24T09:42:13.972 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:13.971+0000 7f38237fe700 1 -- 172.21.15.155:0/1009428200 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 18) v1 ==== 90786+0+0 (secure 0 0 0) 0x7f3828010380 con 0x7f382c194450 2024-03-24T09:42:13.972 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:13.971+0000 7f38237fe700 1 --2- 172.21.15.155:0/1009428200 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f381006cd60 0x7f381006f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:42:13.972 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:13.971+0000 7f38237fe700 1 -- 172.21.15.155:0/1009428200 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f382808ac80 con 0x7f382c194450 2024-03-24T09:42:13.972 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:13.971+0000 7f383217b700 1 --2- 172.21.15.155:0/1009428200 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f381006cd60 0x7f381006f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:42:13.972 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:13.972+0000 7f383217b700 1 --2- 172.21.15.155:0/1009428200 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f381006cd60 0x7f381006f220 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f381c005d10 tx=0x7f381c005c20 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:42:13.976 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:13.977+0000 7f38237fe700 1 -- 172.21.15.155:0/1009428200 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3828056130 con 0x7f382c194450 2024-03-24T09:42:14.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:13 smithi022 ceph-mon[29512]: mgrmap e18: smithi022.emgmxz(active, since 34s), standbys: smithi155.itwzew 2024-03-24T09:42:14.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:13 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mgr metadata", "who": "smithi155.itwzew", "id": "smithi155.itwzew"}]: dispatch 2024-03-24T09:42:14.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:13 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:42:14.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:13 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mon metadata", "id": "smithi155"}]: dispatch 2024-03-24T09:42:14.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:13 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:14.254 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:14.254+0000 7f38343df700 1 -- 172.21.15.155:0/1009428200 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f382c195220 con 0x7f382c194450 2024-03-24T09:42:14.255 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:14.255+0000 7f38237fe700 1 -- 172.21.15.155:0/1009428200 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1022 (secure 0 0 0) 0x7f382808a440 con 0x7f382c194450 2024-03-24T09:42:14.255 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-24T09:42:14.255 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":2,"fsid":"45356448-e9c2-11ee-95cd-87774f69a715","modified":"2024-03-24T09:42:07.465180Z","created":"2024-03-24T09:39:22.550742Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi022","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:3300","nonce":0},{"type":"v1","addr":"172.21.15.22:6789","nonce":0}]},"addr":"172.21.15.22:6789/0","public_addr":"172.21.15.22:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi155","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-24T09:42:14.257 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:14.257+0000 7f38343df700 1 -- 172.21.15.155:0/1009428200 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f381006cd60 msgr2=0x7f381006f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:42:14.257 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:14.257+0000 7f38343df700 1 --2- 172.21.15.155:0/1009428200 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f381006cd60 0x7f381006f220 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f381c005d10 tx=0x7f381c005c20 comp rx=0 tx=0).stop 2024-03-24T09:42:14.257 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:14.257+0000 7f38343df700 1 -- 172.21.15.155:0/1009428200 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f382c194450 msgr2=0x7f382c198800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:42:14.257 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:14.257+0000 7f38343df700 1 --2- 172.21.15.155:0/1009428200 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f382c194450 0x7f382c198800 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f382800b4d0 tx=0x7f382800b890 comp rx=0 tx=0).stop 2024-03-24T09:42:14.257 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:14.257+0000 7f38343df700 1 -- 172.21.15.155:0/1009428200 shutdown_connections 2024-03-24T09:42:14.257 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:14.257+0000 7f38343df700 1 --2- 172.21.15.155:0/1009428200 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f381006cd60 0x7f381006f220 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:42:14.257 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:14.257+0000 7f38343df700 1 --2- 172.21.15.155:0/1009428200 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38180057d0 0x7f382c193f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:42:14.257 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:14.257+0000 7f38343df700 1 --2- 172.21.15.155:0/1009428200 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f382c194450 0x7f382c198800 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:42:14.257 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:14.257+0000 7f38343df700 1 -- 172.21.15.155:0/1009428200 >> 172.21.15.155:0/1009428200 conn(0x7f382c076390 msgr2=0x7f382c0fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:42:14.258 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:14.257+0000 7f38343df700 1 -- 172.21.15.155:0/1009428200 shutdown_connections 2024-03-24T09:42:14.258 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:42:14.257+0000 7f38343df700 1 -- 172.21.15.155:0/1009428200 wait complete. 2024-03-24T09:42:14.258 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 2 2024-03-24T09:42:14.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:14 smithi155 ceph-mon[37016]: from='client.? 172.21.15.155:0/1009428200' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T09:42:14.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:14 smithi155 ceph-mon[37016]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:14.989 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-24T09:42:14.989 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph config generate-minimal-conf 2024-03-24T09:42:15.014 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:14 smithi022 ceph-mon[29512]: from='client.? 172.21.15.155:0/1009428200' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T09:42:15.014 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:14 smithi022 ceph-mon[29512]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:15.227 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:42:15.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:15 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:15.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:15 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:15.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:15 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:42:15.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:15 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:42:15.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:15 smithi155 ceph-mon[37016]: Updating smithi022:/etc/ceph/ceph.conf 2024-03-24T09:42:15.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:15 smithi155 ceph-mon[37016]: Updating smithi155:/etc/ceph/ceph.conf 2024-03-24T09:42:15.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:15 smithi155 ceph-mon[37016]: Updating smithi022:/var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T09:42:15.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:15 smithi155 ceph-mon[37016]: Updating smithi155:/var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T09:42:15.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:15 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:15.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:15 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:15.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:15 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:15.963 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:15 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:15.963 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:15 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:15.963 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:15 smithi155 ceph-mon[37016]: Reconfiguring mon.smithi022 (unknown last config time)... 2024-03-24T09:42:15.963 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:15 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T09:42:15.963 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:15 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T09:42:15.963 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:15 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:42:15.963 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:15 smithi155 ceph-mon[37016]: Reconfiguring daemon mon.smithi022 on smithi022 2024-03-24T09:42:16.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:15 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:16.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:15 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:16.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:15 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:42:16.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:15 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:42:16.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:15 smithi022 ceph-mon[29512]: Updating smithi022:/etc/ceph/ceph.conf 2024-03-24T09:42:16.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:15 smithi022 ceph-mon[29512]: Updating smithi155:/etc/ceph/ceph.conf 2024-03-24T09:42:16.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:15 smithi022 ceph-mon[29512]: Updating smithi022:/var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T09:42:16.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:15 smithi022 ceph-mon[29512]: Updating smithi155:/var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T09:42:16.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:15 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:16.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:15 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:16.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:15 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:16.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:15 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:16.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:15 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:16.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:15 smithi022 ceph-mon[29512]: Reconfiguring mon.smithi022 (unknown last config time)... 2024-03-24T09:42:16.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:15 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T09:42:16.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:15 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T09:42:16.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:15 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:42:16.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:15 smithi022 ceph-mon[29512]: Reconfiguring daemon mon.smithi022 on smithi022 2024-03-24T09:42:16.918 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:16 smithi022 ceph-mon[29512]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:16.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:16 smithi155 ceph-mon[37016]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:17.380 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.380+0000 7f7150387700 1 -- 172.21.15.22:0/219463185 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7148105660 msgr2=0x7f7148105a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:42:17.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.380+0000 7f7150387700 1 --2- 172.21.15.22:0/219463185 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7148105660 0x7f7148105a40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7144009910 tx=0x7f7144009c20 comp rx=0 tx=0).stop 2024-03-24T09:42:17.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.380+0000 7f7150387700 1 -- 172.21.15.22:0/219463185 shutdown_connections 2024-03-24T09:42:17.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.380+0000 7f7150387700 1 --2- 172.21.15.22:0/219463185 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7148105660 0x7f7148105a40 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:42:17.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.380+0000 7f7150387700 1 -- 172.21.15.22:0/219463185 >> 172.21.15.22:0/219463185 conn(0x7f71480fb0e0 msgr2=0x7f71480fd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:42:17.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.380+0000 7f7150387700 1 -- 172.21.15.22:0/219463185 shutdown_connections 2024-03-24T09:42:17.381 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.380+0000 7f7150387700 1 -- 172.21.15.22:0/219463185 wait complete. 2024-03-24T09:42:17.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.381+0000 7f7150387700 1 Processor -- start 2024-03-24T09:42:17.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.381+0000 7f7150387700 1 -- start start 2024-03-24T09:42:17.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.381+0000 7f7150387700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7148105660 0x7f7148196f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:42:17.382 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.381+0000 7f7150387700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7148197450 0x7f71481978d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:42:17.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.381+0000 7f7150387700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714819ff80 con 0x7f7148105660 2024-03-24T09:42:17.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.381+0000 7f7150387700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71481a00c0 con 0x7f7148197450 2024-03-24T09:42:17.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.381+0000 7f714e123700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7148105660 0x7f7148196f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:42:17.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.381+0000 7f714e123700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7148105660 0x7f7148196f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:35846/0 (socket says 172.21.15.22:35846) 2024-03-24T09:42:17.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.381+0000 7f714e123700 1 -- 172.21.15.22:0/2363654817 learned_addr learned my addr 172.21.15.22:0/2363654817 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:42:17.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.382+0000 7f714e123700 1 -- 172.21.15.22:0/2363654817 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7148197450 msgr2=0x7f71481978d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:42:17.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.382+0000 7f714e123700 1 --2- 172.21.15.22:0/2363654817 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7148197450 0x7f71481978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:42:17.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.382+0000 7f714e123700 1 -- 172.21.15.22:0/2363654817 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71440095f0 con 0x7f7148105660 2024-03-24T09:42:17.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.382+0000 7f714e123700 1 --2- 172.21.15.22:0/2363654817 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7148105660 0x7f7148196f10 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7144005b40 tx=0x7f714400bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:42:17.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.382+0000 7f713f7fe700 1 -- 172.21.15.22:0/2363654817 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f714401d070 con 0x7f7148105660 2024-03-24T09:42:17.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.382+0000 7f7150387700 1 -- 172.21.15.22:0/2363654817 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71481a02c0 con 0x7f7148105660 2024-03-24T09:42:17.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.382+0000 7f713f7fe700 1 -- 172.21.15.22:0/2363654817 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7144003d50 con 0x7f7148105660 2024-03-24T09:42:17.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.382+0000 7f713f7fe700 1 -- 172.21.15.22:0/2363654817 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7144017440 con 0x7f7148105660 2024-03-24T09:42:17.383 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.382+0000 7f7150387700 1 -- 172.21.15.22:0/2363654817 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71481a07e0 con 0x7f7148105660 2024-03-24T09:42:17.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.383+0000 7f7150387700 1 -- 172.21.15.22:0/2363654817 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7148100d10 con 0x7f7148105660 2024-03-24T09:42:17.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.384+0000 7f713f7fe700 1 -- 172.21.15.22:0/2363654817 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 18) v1 ==== 90786+0+0 (secure 0 0 0) 0x7f714400f7d0 con 0x7f7148105660 2024-03-24T09:42:17.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.384+0000 7f713f7fe700 1 --2- 172.21.15.22:0/2363654817 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f713406cdb0 0x7f713406f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:42:17.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.384+0000 7f713f7fe700 1 -- 172.21.15.22:0/2363654817 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f714408c390 con 0x7f7148105660 2024-03-24T09:42:17.384 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.384+0000 7f714d922700 1 --2- 172.21.15.22:0/2363654817 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f713406cdb0 0x7f713406f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:42:17.385 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.385+0000 7f714d922700 1 --2- 172.21.15.22:0/2363654817 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f713406cdb0 0x7f713406f270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f71380098e0 tx=0x7f7138006dd0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:42:17.390 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.390+0000 7f713f7fe700 1 -- 172.21.15.22:0/2363654817 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7144057840 con 0x7f7148105660 2024-03-24T09:42:17.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.593+0000 7f7150387700 1 -- 172.21.15.22:0/2363654817 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f71480623e0 con 0x7f7148105660 2024-03-24T09:42:17.594 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.594+0000 7f713f7fe700 1 -- 172.21.15.22:0/2363654817 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+225 (secure 0 0 0) 0x7f714408bb50 con 0x7f7148105660 2024-03-24T09:42:17.594 INFO:teuthology.orchestra.run.smithi022.stdout:# minimal ceph.conf for 45356448-e9c2-11ee-95cd-87774f69a715 2024-03-24T09:42:17.594 INFO:teuthology.orchestra.run.smithi022.stdout:[global] 2024-03-24T09:42:17.594 INFO:teuthology.orchestra.run.smithi022.stdout: fsid = 45356448-e9c2-11ee-95cd-87774f69a715 2024-03-24T09:42:17.594 INFO:teuthology.orchestra.run.smithi022.stdout: mon_host = [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-03-24T09:42:17.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.596+0000 7f7150387700 1 -- 172.21.15.22:0/2363654817 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f713406cdb0 msgr2=0x7f713406f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:42:17.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.596+0000 7f7150387700 1 --2- 172.21.15.22:0/2363654817 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f713406cdb0 0x7f713406f270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f71380098e0 tx=0x7f7138006dd0 comp rx=0 tx=0).stop 2024-03-24T09:42:17.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.596+0000 7f7150387700 1 -- 172.21.15.22:0/2363654817 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7148105660 msgr2=0x7f7148196f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:42:17.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.596+0000 7f7150387700 1 --2- 172.21.15.22:0/2363654817 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7148105660 0x7f7148196f10 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7144005b40 tx=0x7f714400bdc0 comp rx=0 tx=0).stop 2024-03-24T09:42:17.596 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.596+0000 7f7150387700 1 -- 172.21.15.22:0/2363654817 shutdown_connections 2024-03-24T09:42:17.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.596+0000 7f7150387700 1 --2- 172.21.15.22:0/2363654817 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f713406cdb0 0x7f713406f270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:42:17.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.596+0000 7f7150387700 1 --2- 172.21.15.22:0/2363654817 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7148105660 0x7f7148196f10 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:42:17.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.596+0000 7f7150387700 1 --2- 172.21.15.22:0/2363654817 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7148197450 0x7f71481978d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:42:17.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.596+0000 7f7150387700 1 -- 172.21.15.22:0/2363654817 >> 172.21.15.22:0/2363654817 conn(0x7f71480fb0e0 msgr2=0x7f71480fb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:42:17.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.596+0000 7f7150387700 1 -- 172.21.15.22:0/2363654817 shutdown_connections 2024-03-24T09:42:17.597 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:17.596+0000 7f7150387700 1 -- 172.21.15.22:0/2363654817 wait complete. 2024-03-24T09:42:18.448 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-24T09:42:18.449 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T09:42:18.449 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T09:42:18.518 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T09:42:18.518 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T09:42:18.584 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-24T09:42:18.584 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T09:42:18.619 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-24T09:42:18.619 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T09:42:18.689 INFO:tasks.cephadm:Deploying OSDs... 2024-03-24T09:42:18.689 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T09:42:18.689 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T09:42:18.704 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T09:42:18.704 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_1 2024-03-24T09:42:18.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:18 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/2363654817' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:42:18.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:18 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:18.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:18 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:18.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:18 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi022.emgmxz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T09:42:18.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:18 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T09:42:18.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:18 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:42:18.731 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:18 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/2363654817' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:42:18.731 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:18 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:18.731 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:18 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:18.732 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:18 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi022.emgmxz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T09:42:18.732 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:18 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T09:42:18.732 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:18 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:42:18.745 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T09:42:18.745 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T09:42:18.746 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-24T09:42:18.746 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T09:42:18.746 INFO:teuthology.orchestra.run.smithi022.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T09:42:18.746 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2024-03-24 09:41:29.801285267 +0000 2024-03-24T09:42:18.746 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2024-03-24 09:35:44.262394637 +0000 2024-03-24T09:42:18.746 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2024-03-24 09:35:44.262394637 +0000 2024-03-24T09:42:18.746 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: 2024-03-24 09:35:44.262394637 +0000 2024-03-24T09:42:18.746 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T09:42:18.810 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2024-03-24T09:42:18.810 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2024-03-24T09:42:18.811 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 9.5916e-05 s, 5.3 MB/s 2024-03-24T09:42:18.812 DEBUG:teuthology.orchestra.run.smithi022:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T09:42:18.868 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_2 2024-03-24T09:42:18.923 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T09:42:18.923 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T09:42:18.923 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-24T09:42:18.924 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T09:42:18.924 INFO:teuthology.orchestra.run.smithi022.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T09:42:18.924 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2024-03-24 09:41:29.802285254 +0000 2024-03-24T09:42:18.924 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2024-03-24 09:35:44.584390636 +0000 2024-03-24T09:42:18.924 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2024-03-24 09:35:44.584390636 +0000 2024-03-24T09:42:18.924 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: 2024-03-24 09:35:44.584390636 +0000 2024-03-24T09:42:18.924 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T09:42:18.990 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2024-03-24T09:42:18.990 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2024-03-24T09:42:18.991 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 9.8569e-05 s, 5.2 MB/s 2024-03-24T09:42:18.991 DEBUG:teuthology.orchestra.run.smithi022:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T09:42:19.047 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_3 2024-03-24T09:42:19.101 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T09:42:19.101 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T09:42:19.101 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-24T09:42:19.101 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T09:42:19.102 INFO:teuthology.orchestra.run.smithi022.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T09:42:19.102 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2024-03-24 09:41:29.802285254 +0000 2024-03-24T09:42:19.102 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2024-03-24 09:35:44.925386399 +0000 2024-03-24T09:42:19.102 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2024-03-24 09:35:44.925386399 +0000 2024-03-24T09:42:19.102 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: 2024-03-24 09:35:44.925386399 +0000 2024-03-24T09:42:19.102 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T09:42:19.166 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2024-03-24T09:42:19.166 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2024-03-24T09:42:19.166 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 9.8659e-05 s, 5.2 MB/s 2024-03-24T09:42:19.167 DEBUG:teuthology.orchestra.run.smithi022:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T09:42:19.223 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_4 2024-03-24T09:42:19.278 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T09:42:19.278 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T09:42:19.278 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-24T09:42:19.278 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T09:42:19.278 INFO:teuthology.orchestra.run.smithi022.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T09:42:19.279 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2024-03-24 09:41:29.802285254 +0000 2024-03-24T09:42:19.279 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2024-03-24 09:35:45.253382323 +0000 2024-03-24T09:42:19.279 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2024-03-24 09:35:45.253382323 +0000 2024-03-24T09:42:19.279 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: 2024-03-24 09:35:45.253382323 +0000 2024-03-24T09:42:19.279 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T09:42:19.343 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2024-03-24T09:42:19.343 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2024-03-24T09:42:19.343 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 9.999e-05 s, 5.1 MB/s 2024-03-24T09:42:19.345 DEBUG:teuthology.orchestra.run.smithi022:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T09:42:19.400 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-24T09:42:19.400 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T09:42:19.419 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T09:42:19.419 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_1 2024-03-24T09:42:19.478 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T09:42:19.479 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T09:42:19.479 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-24T09:42:19.479 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T09:42:19.479 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T09:42:19.479 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-03-24 09:42:06.458319092 +0000 2024-03-24T09:42:19.479 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-03-24 09:35:41.859409768 +0000 2024-03-24T09:42:19.479 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-03-24 09:35:41.859409768 +0000 2024-03-24T09:42:19.479 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-03-24 09:35:41.859409768 +0000 2024-03-24T09:42:19.479 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T09:42:19.548 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-03-24T09:42:19.548 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-03-24T09:42:19.548 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000262206 s, 2.0 MB/s 2024-03-24T09:42:19.550 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T09:42:19.609 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_2 2024-03-24T09:42:19.639 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:19 smithi022 ceph-mon[29512]: Reconfiguring mgr.smithi022.emgmxz (unknown last config time)... 2024-03-24T09:42:19.639 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:19 smithi022 ceph-mon[29512]: Reconfiguring daemon mgr.smithi022.emgmxz on smithi022 2024-03-24T09:42:19.639 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:19 smithi022 ceph-mon[29512]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:19.667 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T09:42:19.667 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T09:42:19.667 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-24T09:42:19.667 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T09:42:19.668 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T09:42:19.668 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-03-24 09:42:06.458319092 +0000 2024-03-24T09:42:19.668 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-03-24 09:35:42.200402375 +0000 2024-03-24T09:42:19.668 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-03-24 09:35:42.200402375 +0000 2024-03-24T09:42:19.668 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-03-24 09:35:42.200402375 +0000 2024-03-24T09:42:19.668 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T09:42:19.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:19 smithi155 ceph-mon[37016]: Reconfiguring mgr.smithi022.emgmxz (unknown last config time)... 2024-03-24T09:42:19.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:19 smithi155 ceph-mon[37016]: Reconfiguring daemon mgr.smithi022.emgmxz on smithi022 2024-03-24T09:42:19.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:19 smithi155 ceph-mon[37016]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:19.736 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-03-24T09:42:19.736 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-03-24T09:42:19.736 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000229682 s, 2.2 MB/s 2024-03-24T09:42:19.737 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T09:42:19.795 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_3 2024-03-24T09:42:19.851 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T09:42:19.851 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T09:42:19.852 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-24T09:42:19.852 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T09:42:19.852 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T09:42:19.852 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-03-24 09:42:06.458319092 +0000 2024-03-24T09:42:19.852 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-03-24 09:35:42.537395068 +0000 2024-03-24T09:42:19.852 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-03-24 09:35:42.537395068 +0000 2024-03-24T09:42:19.852 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-03-24 09:35:42.537395068 +0000 2024-03-24T09:42:19.852 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T09:42:19.918 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-03-24T09:42:19.919 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-03-24T09:42:19.919 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000239228 s, 2.1 MB/s 2024-03-24T09:42:19.920 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T09:42:19.978 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_4 2024-03-24T09:42:20.034 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T09:42:20.034 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T09:42:20.034 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-24T09:42:20.034 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T09:42:20.034 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T09:42:20.035 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-03-24 09:42:06.459319071 +0000 2024-03-24T09:42:20.035 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-03-24 09:35:42.879387653 +0000 2024-03-24T09:42:20.035 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-03-24 09:35:42.879387653 +0000 2024-03-24T09:42:20.035 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-03-24 09:35:42.879387653 +0000 2024-03-24T09:42:20.035 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T09:42:20.101 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-03-24T09:42:20.101 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-03-24T09:42:20.101 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000211607 s, 2.4 MB/s 2024-03-24T09:42:20.102 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T09:42:20.159 INFO:tasks.cephadm:Deploying osd.0 on smithi022 with /dev/vg_nvme/lv_4... 2024-03-24T09:42:20.159 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-24T09:42:20.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:20 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:20.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:20 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:20.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:20 smithi022 ceph-mon[29512]: Reconfiguring ceph-exporter.smithi022 (monmap changed)... 2024-03-24T09:42:20.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:20 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi022", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-24T09:42:20.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:20 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:42:20.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:20 smithi022 ceph-mon[29512]: Reconfiguring daemon ceph-exporter.smithi022 on smithi022 2024-03-24T09:42:20.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:20 smithi022 ceph-mon[29512]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:20.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:20 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:20.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:20 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:20.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:20 smithi155 ceph-mon[37016]: Reconfiguring ceph-exporter.smithi022 (monmap changed)... 2024-03-24T09:42:20.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:20 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi022", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-24T09:42:20.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:20 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:42:20.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:20 smithi155 ceph-mon[37016]: Reconfiguring daemon ceph-exporter.smithi022 on smithi022 2024-03-24T09:42:20.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:20 smithi155 ceph-mon[37016]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:21.132 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:42:22.255 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:22 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:22.256 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:22 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:22.256 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:22 smithi022 ceph-mon[29512]: Reconfiguring crash.smithi022 (monmap changed)... 2024-03-24T09:42:22.256 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:22 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi022", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T09:42:22.256 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:22 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:42:22.256 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:22 smithi022 ceph-mon[29512]: Reconfiguring daemon crash.smithi022 on smithi022 2024-03-24T09:42:22.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:22 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:22.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:22 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:22.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:22 smithi155 ceph-mon[37016]: Reconfiguring crash.smithi022 (monmap changed)... 2024-03-24T09:42:22.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:22 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi022", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T09:42:22.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:22 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:42:22.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:22 smithi155 ceph-mon[37016]: Reconfiguring daemon crash.smithi022 on smithi022 2024-03-24T09:42:23.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:23 smithi155 ceph-mon[37016]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:23.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:23 smithi022 ceph-mon[29512]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:24.148 INFO:teuthology.orchestra.run.smithi022.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-24T09:42:24.148 INFO:teuthology.orchestra.run.smithi022.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-24T09:42:24.149 INFO:teuthology.orchestra.run.smithi022.stderr: stderr: 10+0 records in 2024-03-24T09:42:24.149 INFO:teuthology.orchestra.run.smithi022.stderr:10+0 records out 2024-03-24T09:42:24.149 INFO:teuthology.orchestra.run.smithi022.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00835895 s, 1.3 GB/s 2024-03-24T09:42:24.149 INFO:teuthology.orchestra.run.smithi022.stderr:--> Zapping successful for: 2024-03-24T09:42:24.399 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:24 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:42:24.400 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:24 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:24.400 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:24 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:24.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:24 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:42:24.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:24 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:24.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:24 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:24.986 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi022:vg_nvme/lv_4 2024-03-24T09:42:25.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:25 smithi155 ceph-mon[37016]: Reconfiguring alertmanager.smithi022 (dependencies changed)... 2024-03-24T09:42:25.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:25 smithi155 ceph-mon[37016]: Reconfiguring daemon alertmanager.smithi022 on smithi022 2024-03-24T09:42:25.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:25 smithi155 ceph-mon[37016]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:25.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:25 smithi022 ceph-mon[29512]: Reconfiguring alertmanager.smithi022 (dependencies changed)... 2024-03-24T09:42:25.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:25 smithi022 ceph-mon[29512]: Reconfiguring daemon alertmanager.smithi022 on smithi022 2024-03-24T09:42:25.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:25 smithi022 ceph-mon[29512]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:25.968 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:42:27.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:27 smithi022 ceph-mon[29512]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:27.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:27 smithi155 ceph-mon[37016]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:28.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:28.885+0000 7f6411251700 1 -- 172.21.15.22:0/4012166331 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f640c0fedf0 msgr2=0x7f640c0ff210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:42:28.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:28.885+0000 7f6411251700 1 --2- 172.21.15.22:0/4012166331 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f640c0fedf0 0x7f640c0ff210 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f63fc00b0a0 tx=0x7f63fc00b3b0 comp rx=0 tx=0).stop 2024-03-24T09:42:28.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:28.885+0000 7f6411251700 1 -- 172.21.15.22:0/4012166331 shutdown_connections 2024-03-24T09:42:28.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:28.885+0000 7f6411251700 1 --2- 172.21.15.22:0/4012166331 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f640c101e70 0x7f640c1022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:42:28.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:28.885+0000 7f6411251700 1 --2- 172.21.15.22:0/4012166331 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f640c0fedf0 0x7f640c0ff210 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:42:28.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:28.885+0000 7f6411251700 1 -- 172.21.15.22:0/4012166331 >> 172.21.15.22:0/4012166331 conn(0x7f640c0fa730 msgr2=0x7f640c0fcb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:42:28.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:28.886+0000 7f6411251700 1 -- 172.21.15.22:0/4012166331 shutdown_connections 2024-03-24T09:42:28.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:28.886+0000 7f6411251700 1 -- 172.21.15.22:0/4012166331 wait complete. 2024-03-24T09:42:28.888 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:28.886+0000 7f6411251700 1 Processor -- start 2024-03-24T09:42:28.888 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:28.886+0000 7f6411251700 1 -- start start 2024-03-24T09:42:28.888 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:28.887+0000 7f6411251700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f640c0fedf0 0x7f640c101710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:42:28.888 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:28.887+0000 7f6411251700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f640c101e70 0x7f640c0ffe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:42:28.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:28.887+0000 7f6411251700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640c101d30 con 0x7f640c101e70 2024-03-24T09:42:28.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:28.887+0000 7f6411251700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640c100360 con 0x7f640c0fedf0 2024-03-24T09:42:28.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:28.887+0000 7f640a59c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f640c101e70 0x7f640c0ffe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:42:28.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:28.887+0000 7f640a59c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f640c101e70 0x7f640c0ffe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:60050/0 (socket says 172.21.15.22:60050) 2024-03-24T09:42:28.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:28.887+0000 7f640a59c700 1 -- 172.21.15.22:0/2604678711 learned_addr learned my addr 172.21.15.22:0/2604678711 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:42:28.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:28.887+0000 7f640ad9d700 1 --2- 172.21.15.22:0/2604678711 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f640c0fedf0 0x7f640c101710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:42:28.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:28.887+0000 7f640a59c700 1 -- 172.21.15.22:0/2604678711 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f640c0fedf0 msgr2=0x7f640c101710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:42:28.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:28.887+0000 7f640a59c700 1 --2- 172.21.15.22:0/2604678711 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f640c0fedf0 0x7f640c101710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:42:28.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:28.887+0000 7f640a59c700 1 -- 172.21.15.22:0/2604678711 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63fc009d00 con 0x7f640c101e70 2024-03-24T09:42:28.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:28.887+0000 7f640ad9d700 1 --2- 172.21.15.22:0/2604678711 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f640c0fedf0 0x7f640c101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T09:42:28.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:28.887+0000 7f640a59c700 1 --2- 172.21.15.22:0/2604678711 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f640c101e70 0x7f640c0ffe20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f640000b4b0 tx=0x7f640000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:42:28.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:28.888+0000 7f63f3fff700 1 -- 172.21.15.22:0/2604678711 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6400011800 con 0x7f640c101e70 2024-03-24T09:42:28.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:28.888+0000 7f6411251700 1 -- 172.21.15.22:0/2604678711 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f640c100640 con 0x7f640c101e70 2024-03-24T09:42:28.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:28.888+0000 7f63f3fff700 1 -- 172.21.15.22:0/2604678711 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6400011e20 con 0x7f640c101e70 2024-03-24T09:42:28.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:28.888+0000 7f63f3fff700 1 -- 172.21.15.22:0/2604678711 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f640000f2e0 con 0x7f640c101e70 2024-03-24T09:42:28.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:28.888+0000 7f6411251700 1 -- 172.21.15.22:0/2604678711 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f640c100c10 con 0x7f640c101e70 2024-03-24T09:42:28.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:28.889+0000 7f6411251700 1 -- 172.21.15.22:0/2604678711 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f640c04ea80 con 0x7f640c101e70 2024-03-24T09:42:28.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:28.889+0000 7f63f3fff700 1 -- 172.21.15.22:0/2604678711 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 18) v1 ==== 90786+0+0 (secure 0 0 0) 0x7f6400011960 con 0x7f640c101e70 2024-03-24T09:42:28.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:28.890+0000 7f63f3fff700 1 --2- 172.21.15.22:0/2604678711 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f63f406cc60 0x7f63f406f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:42:28.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:28.890+0000 7f63f3fff700 1 -- 172.21.15.22:0/2604678711 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f640008ad30 con 0x7f640c101e70 2024-03-24T09:42:28.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:28.890+0000 7f640ad9d700 1 --2- 172.21.15.22:0/2604678711 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f63f406cc60 0x7f63f406f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:42:28.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:28.890+0000 7f640ad9d700 1 --2- 172.21.15.22:0/2604678711 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f63f406cc60 0x7f63f406f120 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f63fc009480 tx=0x7f63fc007910 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:42:28.897 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:28.897+0000 7f63f3fff700 1 -- 172.21.15.22:0/2604678711 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6400059c70 con 0x7f640c101e70 2024-03-24T09:42:29.121 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:29.121+0000 7f6411251700 1 -- 172.21.15.22:0/2604678711 --> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f640c19e700 con 0x7f63f406cc60 2024-03-24T09:42:29.650 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:29 smithi022 ceph-mon[29512]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:29.650 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:29 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:42:29.650 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:29 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:42:29.650 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:29 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:42:29.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:29 smithi155 ceph-mon[37016]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:29.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:29 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:42:29.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:29 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:42:29.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:29 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:42:30.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:30 smithi022 ceph-mon[29512]: from='client.14254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:42:30.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:30 smithi155 ceph-mon[37016]: from='client.14254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:42:31.738 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:31 smithi022 ceph-mon[29512]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:31.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:31 smithi155 ceph-mon[37016]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:32.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:32 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:32.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:32 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:32.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:32 smithi155 ceph-mon[37016]: Reconfiguring grafana.smithi022 (dependencies changed)... 2024-03-24T09:42:32.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:32 smithi155 ceph-mon[37016]: Reconfiguring daemon grafana.smithi022 on smithi022 2024-03-24T09:42:32.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:32 smithi155 ceph-mon[37016]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:33.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:32 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:33.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:32 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:33.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:32 smithi022 ceph-mon[29512]: Reconfiguring grafana.smithi022 (dependencies changed)... 2024-03-24T09:42:33.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:32 smithi022 ceph-mon[29512]: Reconfiguring daemon grafana.smithi022 on smithi022 2024-03-24T09:42:33.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:32 smithi022 ceph-mon[29512]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:33.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:33 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/1752573569' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba411044-c790-44b5-881e-4e86675a868b"}]: dispatch 2024-03-24T09:42:33.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:33 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/1752573569' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba411044-c790-44b5-881e-4e86675a868b"}]': finished 2024-03-24T09:42:33.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:33 smithi022 ceph-mon[29512]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T09:42:33.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:33 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T09:42:33.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:33 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/2942704674' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:42:33.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:33 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/1752573569' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba411044-c790-44b5-881e-4e86675a868b"}]: dispatch 2024-03-24T09:42:33.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:33 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/1752573569' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba411044-c790-44b5-881e-4e86675a868b"}]': finished 2024-03-24T09:42:33.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:33 smithi155 ceph-mon[37016]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T09:42:33.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:33 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T09:42:33.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:33 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/2942704674' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:42:34.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:34 smithi155 ceph-mon[37016]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:35.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:34 smithi022 ceph-mon[29512]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:37.777 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:37 smithi022 ceph-mon[29512]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:37.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:37 smithi155 ceph-mon[37016]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:38.683 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:38 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:42:38.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:38 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:42:39.714 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:39 smithi022 ceph-mon[29512]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:39.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:39 smithi155 ceph-mon[37016]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:41.446 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:41 smithi022 ceph-mon[29512]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:41.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:41 smithi155 ceph-mon[37016]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:42.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:42 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:42.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:42 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:42.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:42 smithi155 ceph-mon[37016]: Reconfiguring prometheus.smithi022 (dependencies changed)... 2024-03-24T09:42:42.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:42 smithi155 ceph-mon[37016]: Reconfiguring daemon prometheus.smithi022 on smithi022 2024-03-24T09:42:42.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:42 smithi155 ceph-mon[37016]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:43.010 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:42 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:43.010 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:42 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:43.010 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:42 smithi022 ceph-mon[29512]: Reconfiguring prometheus.smithi022 (dependencies changed)... 2024-03-24T09:42:43.010 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:42 smithi022 ceph-mon[29512]: Reconfiguring daemon prometheus.smithi022 on smithi022 2024-03-24T09:42:43.010 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:42 smithi022 ceph-mon[29512]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:43.993 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:43 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T09:42:43.994 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:43 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:42:43.994 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:43 smithi022 ceph-mon[29512]: Deploying daemon osd.0 on smithi022 2024-03-24T09:42:44.211 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:43 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T09:42:44.211 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:43 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:42:44.211 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:43 smithi155 ceph-mon[37016]: Deploying daemon osd.0 on smithi022 2024-03-24T09:42:45.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:44 smithi022 ceph-mon[29512]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:45.211 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:44 smithi155 ceph-mon[37016]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:47.561 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:47 smithi022 ceph-mon[29512]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:47.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:47 smithi155 ceph-mon[37016]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:48.861 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:48 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:48.861 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:48 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:48.862 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:48 smithi155 ceph-mon[37016]: Reconfiguring ceph-exporter.smithi155 (monmap changed)... 2024-03-24T09:42:48.862 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:48 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi155", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-24T09:42:48.862 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:48 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:42:48.862 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:48 smithi155 ceph-mon[37016]: Reconfiguring daemon ceph-exporter.smithi155 on smithi155 2024-03-24T09:42:48.862 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:48 smithi155 ceph-mon[37016]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:48.951 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:48 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:48.952 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:48 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:48.952 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:48 smithi022 ceph-mon[29512]: Reconfiguring ceph-exporter.smithi155 (monmap changed)... 2024-03-24T09:42:48.952 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:48 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi155", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-24T09:42:48.952 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:48 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:42:48.952 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:48 smithi022 ceph-mon[29512]: Reconfiguring daemon ceph-exporter.smithi155 on smithi155 2024-03-24T09:42:48.952 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:48 smithi022 ceph-mon[29512]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:50.191 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:50 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:50.191 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:50 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:50.192 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:50 smithi022 ceph-mon[29512]: Reconfiguring crash.smithi155 (monmap changed)... 2024-03-24T09:42:50.192 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:50 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi155", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T09:42:50.192 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:50 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:42:50.192 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:50 smithi022 ceph-mon[29512]: Reconfiguring daemon crash.smithi155 on smithi155 2024-03-24T09:42:50.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:50 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:50.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:50 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:50.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:50 smithi155 ceph-mon[37016]: Reconfiguring crash.smithi155 (monmap changed)... 2024-03-24T09:42:50.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:50 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi155", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T09:42:50.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:50 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:42:50.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:50 smithi155 ceph-mon[37016]: Reconfiguring daemon crash.smithi155 on smithi155 2024-03-24T09:42:51.665 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:51 smithi155 ceph-mon[37016]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:51.665 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:51 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:51.665 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:51 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:51.665 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:51 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi155.itwzew", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T09:42:51.665 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:51 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T09:42:51.665 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:51 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:42:52.016 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:51 smithi022 ceph-mon[29512]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:52.017 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:51 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:52.017 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:51 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:52.017 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:51 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi155.itwzew", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T09:42:52.017 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:51 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T09:42:52.017 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:51 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:42:52.558 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:52 smithi022 ceph-mon[29512]: Reconfiguring mgr.smithi155.itwzew (monmap changed)... 2024-03-24T09:42:52.558 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:52 smithi022 ceph-mon[29512]: Reconfiguring daemon mgr.smithi155.itwzew on smithi155 2024-03-24T09:42:52.559 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:52 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:52.559 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:52 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:52.559 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:52 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T09:42:52.559 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:52 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T09:42:52.559 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:52 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:42:52.702 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:52 smithi155 ceph-mon[37016]: Reconfiguring mgr.smithi155.itwzew (monmap changed)... 2024-03-24T09:42:52.702 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:52 smithi155 ceph-mon[37016]: Reconfiguring daemon mgr.smithi155.itwzew on smithi155 2024-03-24T09:42:52.702 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:52 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:52.703 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:52 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:52.703 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:52 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T09:42:52.703 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:52 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T09:42:52.703 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:52 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:42:53.709 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:53 smithi022 ceph-mon[29512]: Reconfiguring mon.smithi155 (monmap changed)... 2024-03-24T09:42:53.709 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:53 smithi022 ceph-mon[29512]: Reconfiguring daemon mon.smithi155 on smithi155 2024-03-24T09:42:53.709 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:53 smithi022 ceph-mon[29512]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:53.709 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:53 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:53.710 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:53 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:53.710 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:53 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:42:53.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:53 smithi155 ceph-mon[37016]: Reconfiguring mon.smithi155 (monmap changed)... 2024-03-24T09:42:53.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:53 smithi155 ceph-mon[37016]: Reconfiguring daemon mon.smithi155 on smithi155 2024-03-24T09:42:53.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:53 smithi155 ceph-mon[37016]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:53.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:53 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:53.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:53 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:53.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:53 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:42:54.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:54 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:54.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:54 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:54.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:54 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T09:42:54.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:54 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi022.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T09:42:54.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:54 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:54.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:54 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T09:42:54.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:54 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi022.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T09:42:54.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:54 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:54.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:54 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T09:42:54.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:54 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi022.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T09:42:54.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:54 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:54.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:54 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:42:54.901 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:54 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:54.901 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:54 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:54.901 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:54 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T09:42:54.901 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:54 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi022.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T09:42:54.901 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:54 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:54.901 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:54 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T09:42:54.901 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:54 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi022.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T09:42:54.901 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:54 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:54.901 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:54 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T09:42:54.902 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:54 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi022.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T09:42:54.902 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:54 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:54.902 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:54 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:42:55.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:55 smithi022 ceph-mon[29512]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T09:42:55.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:55 smithi022 ceph-mon[29512]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi022.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T09:42:55.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:55 smithi022 ceph-mon[29512]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T09:42:55.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:55 smithi022 ceph-mon[29512]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi022.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T09:42:55.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:55 smithi022 ceph-mon[29512]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T09:42:55.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:55 smithi022 ceph-mon[29512]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi022.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T09:42:55.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:55 smithi022 ceph-mon[29512]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:55.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:55 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:55.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:55 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:55.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:55 smithi155 ceph-mon[37016]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T09:42:55.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:55 smithi155 ceph-mon[37016]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi022.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T09:42:55.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:55 smithi155 ceph-mon[37016]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T09:42:55.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:55 smithi155 ceph-mon[37016]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi022.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T09:42:55.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:55 smithi155 ceph-mon[37016]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T09:42:55.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:55 smithi155 ceph-mon[37016]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi022.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T09:42:55.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:55 smithi155 ceph-mon[37016]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:55.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:55 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:55.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:55 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:56.620 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:56.620+0000 7f63f3fff700 1 -- 172.21.15.22:0/2604678711 <== mgr.14215 v2:172.21.15.22:6800/773863266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f640c19e700 con 0x7f63f406cc60 2024-03-24T09:42:56.621 INFO:teuthology.orchestra.run.smithi022.stdout:Created osd(s) 0 on host 'smithi022' 2024-03-24T09:42:56.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:56.622+0000 7f6411251700 1 -- 172.21.15.22:0/2604678711 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f63f406cc60 msgr2=0x7f63f406f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:42:56.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:56.622+0000 7f6411251700 1 --2- 172.21.15.22:0/2604678711 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f63f406cc60 0x7f63f406f120 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f63fc009480 tx=0x7f63fc007910 comp rx=0 tx=0).stop 2024-03-24T09:42:56.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:56.622+0000 7f6411251700 1 -- 172.21.15.22:0/2604678711 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f640c101e70 msgr2=0x7f640c0ffe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:42:56.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:56.622+0000 7f6411251700 1 --2- 172.21.15.22:0/2604678711 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f640c101e70 0x7f640c0ffe20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f640000b4b0 tx=0x7f640000b870 comp rx=0 tx=0).stop 2024-03-24T09:42:56.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:56.622+0000 7f6411251700 1 -- 172.21.15.22:0/2604678711 shutdown_connections 2024-03-24T09:42:56.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:56.622+0000 7f6411251700 1 --2- 172.21.15.22:0/2604678711 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f63f406cc60 0x7f63f406f120 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:42:56.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:56.622+0000 7f6411251700 1 --2- 172.21.15.22:0/2604678711 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f640c0fedf0 0x7f640c101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:42:56.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:56.622+0000 7f6411251700 1 --2- 172.21.15.22:0/2604678711 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f640c101e70 0x7f640c0ffe20 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:42:56.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:56.622+0000 7f6411251700 1 -- 172.21.15.22:0/2604678711 >> 172.21.15.22:0/2604678711 conn(0x7f640c0fa730 msgr2=0x7f640c1050a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:42:56.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:56.622+0000 7f6411251700 1 -- 172.21.15.22:0/2604678711 shutdown_connections 2024-03-24T09:42:56.624 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:42:56.622+0000 7f6411251700 1 -- 172.21.15.22:0/2604678711 wait complete. 2024-03-24T09:42:57.493 DEBUG:teuthology.orchestra.run.smithi022:osd.0> sudo journalctl -f -n 0 -u ceph-45356448-e9c2-11ee-95cd-87774f69a715@osd.0.service 2024-03-24T09:42:57.496 INFO:tasks.cephadm:Deploying osd.1 on smithi022 with /dev/vg_nvme/lv_3... 2024-03-24T09:42:57.496 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-24T09:42:57.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:57 smithi022 ceph-mon[29512]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:57.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:57 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:57.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:57 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:57.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:57 smithi022 ceph-mon[29512]: from='osd.0 [v2:172.21.15.22:6802/3431970421,v1:172.21.15.22:6803/3431970421]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T09:42:57.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:57 smithi155 ceph-mon[37016]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:57.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:57 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:57.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:57 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:42:57.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:57 smithi155 ceph-mon[37016]: from='osd.0 [v2:172.21.15.22:6802/3431970421,v1:172.21.15.22:6803/3431970421]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T09:42:58.256 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:42:58.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:58 smithi022 ceph-mon[29512]: from='osd.0 [v2:172.21.15.22:6802/3431970421,v1:172.21.15.22:6803/3431970421]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T09:42:58.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:58 smithi022 ceph-mon[29512]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T09:42:58.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:58 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T09:42:58.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:58 smithi022 ceph-mon[29512]: from='osd.0 [v2:172.21.15.22:6802/3431970421,v1:172.21.15.22:6803/3431970421]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2024-03-24T09:42:58.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:58 smithi022 ceph-mon[29512]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:58.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:42:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:42:58.626+0000 7f897bd56700 -1 osd.0 0 waiting for initial osdmap 2024-03-24T09:42:58.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:42:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:42:58.629+0000 7f897834c700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T09:42:58.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:58 smithi155 ceph-mon[37016]: from='osd.0 [v2:172.21.15.22:6802/3431970421,v1:172.21.15.22:6803/3431970421]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T09:42:58.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:58 smithi155 ceph-mon[37016]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T09:42:58.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:58 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T09:42:58.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:58 smithi155 ceph-mon[37016]: from='osd.0 [v2:172.21.15.22:6802/3431970421,v1:172.21.15.22:6803/3431970421]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2024-03-24T09:42:58.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:58 smithi155 ceph-mon[37016]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:42:59.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:59 smithi022 ceph-mon[29512]: from='osd.0 [v2:172.21.15.22:6802/3431970421,v1:172.21.15.22:6803/3431970421]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]': finished 2024-03-24T09:42:59.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:59 smithi022 ceph-mon[29512]: osdmap e8: 1 total, 0 up, 1 in 2024-03-24T09:42:59.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:59 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T09:42:59.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:42:59 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T09:42:59.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:59 smithi155 ceph-mon[37016]: from='osd.0 [v2:172.21.15.22:6802/3431970421,v1:172.21.15.22:6803/3431970421]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]': finished 2024-03-24T09:42:59.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:59 smithi155 ceph-mon[37016]: osdmap e8: 1 total, 0 up, 1 in 2024-03-24T09:42:59.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:59 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T09:42:59.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:42:59 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T09:43:00.709 INFO:teuthology.orchestra.run.smithi022.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-24T09:43:00.710 INFO:teuthology.orchestra.run.smithi022.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-24T09:43:00.710 INFO:teuthology.orchestra.run.smithi022.stderr: stderr: 10+0 records in 2024-03-24T09:43:00.710 INFO:teuthology.orchestra.run.smithi022.stderr:10+0 records out 2024-03-24T09:43:00.710 INFO:teuthology.orchestra.run.smithi022.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0105336 s, 995 MB/s 2024-03-24T09:43:00.710 INFO:teuthology.orchestra.run.smithi022.stderr:--> Zapping successful for: 2024-03-24T09:43:00.912 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:00 smithi022 ceph-mon[29512]: purged_snaps scrub starts 2024-03-24T09:43:00.912 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:00 smithi022 ceph-mon[29512]: purged_snaps scrub ok 2024-03-24T09:43:00.912 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:00 smithi022 ceph-mon[29512]: osd.0 [v2:172.21.15.22:6802/3431970421,v1:172.21.15.22:6803/3431970421] boot 2024-03-24T09:43:00.912 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:00 smithi022 ceph-mon[29512]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T09:43:00.912 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:00 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T09:43:00.912 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:00 smithi022 ceph-mon[29512]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:43:00.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:00 smithi155 ceph-mon[37016]: purged_snaps scrub starts 2024-03-24T09:43:00.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:00 smithi155 ceph-mon[37016]: purged_snaps scrub ok 2024-03-24T09:43:00.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:00 smithi155 ceph-mon[37016]: osd.0 [v2:172.21.15.22:6802/3431970421,v1:172.21.15.22:6803/3431970421] boot 2024-03-24T09:43:00.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:00 smithi155 ceph-mon[37016]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T09:43:00.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:00 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T09:43:00.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:00 smithi155 ceph-mon[37016]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:43:01.542 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi022:vg_nvme/lv_3 2024-03-24T09:43:01.758 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:43:01.880 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:01 smithi022 ceph-mon[29512]: osdmap e10: 1 total, 1 up, 1 in 2024-03-24T09:43:01.880 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:01 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:01.881 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:01 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:01.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:01 smithi155 ceph-mon[37016]: osdmap e10: 1 total, 1 up, 1 in 2024-03-24T09:43:01.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:01 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:01.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:01 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:02.889 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:02 smithi022 ceph-mon[29512]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:43:02.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:02 smithi155 ceph-mon[37016]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:43:03.346 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:03.345+0000 7fc99c314700 1 -- 172.21.15.22:0/443193262 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc994100bd0 msgr2=0x7fc994101030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:43:03.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:03.345+0000 7fc99c314700 1 --2- 172.21.15.22:0/443193262 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc994100bd0 0x7fc994101030 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc990009910 tx=0x7fc990009c20 comp rx=0 tx=0).stop 2024-03-24T09:43:03.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:03.345+0000 7fc99c314700 1 -- 172.21.15.22:0/443193262 shutdown_connections 2024-03-24T09:43:03.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:03.345+0000 7fc99c314700 1 --2- 172.21.15.22:0/443193262 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc994100bd0 0x7fc994101030 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:43:03.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:03.345+0000 7fc99c314700 1 --2- 172.21.15.22:0/443193262 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9940ff9d0 0x7fc9940ffdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:43:03.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:03.345+0000 7fc99c314700 1 -- 172.21.15.22:0/443193262 >> 172.21.15.22:0/443193262 conn(0x7fc9940faf70 msgr2=0x7fc9940fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:43:03.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:03.345+0000 7fc99c314700 1 -- 172.21.15.22:0/443193262 shutdown_connections 2024-03-24T09:43:03.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:03.345+0000 7fc99c314700 1 -- 172.21.15.22:0/443193262 wait complete. 2024-03-24T09:43:03.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:03.346+0000 7fc99c314700 1 Processor -- start 2024-03-24T09:43:03.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:03.346+0000 7fc99c314700 1 -- start start 2024-03-24T09:43:03.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:03.346+0000 7fc99c314700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc9940ff9d0 0x7fc9941038c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:43:03.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:03.346+0000 7fc99c314700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc994100bd0 0x7fc994101fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:43:03.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:03.346+0000 7fc99c314700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9941024f0 con 0x7fc9940ff9d0 2024-03-24T09:43:03.347 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:03.346+0000 7fc99c314700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc994102630 con 0x7fc994100bd0 2024-03-24T09:43:03.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:03.346+0000 7fc99a0b0700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc9940ff9d0 0x7fc9941038c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:43:03.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:03.346+0000 7fc99a0b0700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc9940ff9d0 0x7fc9941038c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:52988/0 (socket says 172.21.15.22:52988) 2024-03-24T09:43:03.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:03.346+0000 7fc99a0b0700 1 -- 172.21.15.22:0/4186955891 learned_addr learned my addr 172.21.15.22:0/4186955891 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:43:03.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:03.346+0000 7fc9998af700 1 --2- 172.21.15.22:0/4186955891 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc994100bd0 0x7fc994101fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:43:03.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:03.346+0000 7fc99a0b0700 1 -- 172.21.15.22:0/4186955891 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc994100bd0 msgr2=0x7fc994101fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:43:03.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:03.346+0000 7fc99a0b0700 1 --2- 172.21.15.22:0/4186955891 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc994100bd0 0x7fc994101fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:43:03.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:03.346+0000 7fc99a0b0700 1 -- 172.21.15.22:0/4186955891 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9900095f0 con 0x7fc9940ff9d0 2024-03-24T09:43:03.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:03.347+0000 7fc99a0b0700 1 --2- 172.21.15.22:0/4186955891 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc9940ff9d0 0x7fc9941038c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc98400d900 tx=0x7fc98400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:43:03.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:03.347+0000 7fc98b7fe700 1 -- 172.21.15.22:0/4186955891 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9840041f0 con 0x7fc9940ff9d0 2024-03-24T09:43:03.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:03.347+0000 7fc98b7fe700 1 -- 172.21.15.22:0/4186955891 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc984004350 con 0x7fc9940ff9d0 2024-03-24T09:43:03.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:03.347+0000 7fc99c314700 1 -- 172.21.15.22:0/4186955891 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc994102910 con 0x7fc9940ff9d0 2024-03-24T09:43:03.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:03.347+0000 7fc98b7fe700 1 -- 172.21.15.22:0/4186955891 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc984003cf0 con 0x7fc9940ff9d0 2024-03-24T09:43:03.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:03.347+0000 7fc99c314700 1 -- 172.21.15.22:0/4186955891 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc99419e550 con 0x7fc9940ff9d0 2024-03-24T09:43:03.349 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:03.348+0000 7fc99c314700 1 -- 172.21.15.22:0/4186955891 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc99404ea80 con 0x7fc9940ff9d0 2024-03-24T09:43:03.349 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:03.348+0000 7fc98b7fe700 1 -- 172.21.15.22:0/4186955891 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 18) v1 ==== 90786+0+0 (secure 0 0 0) 0x7fc984009730 con 0x7fc9940ff9d0 2024-03-24T09:43:03.349 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:03.349+0000 7fc98b7fe700 1 --2- 172.21.15.22:0/4186955891 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fc98006cc60 0x7fc98006f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:43:03.349 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:03.349+0000 7fc98b7fe700 1 -- 172.21.15.22:0/4186955891 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1982+0+0 (secure 0 0 0) 0x7fc98408b200 con 0x7fc9940ff9d0 2024-03-24T09:43:03.349 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:03.349+0000 7fc9998af700 1 --2- 172.21.15.22:0/4186955891 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fc98006cc60 0x7fc98006f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:43:03.350 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:03.349+0000 7fc9998af700 1 --2- 172.21.15.22:0/4186955891 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fc98006cc60 0x7fc98006f120 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc990005090 tx=0x7fc990005960 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:43:03.352 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:03.353+0000 7fc98b7fe700 1 -- 172.21.15.22:0/4186955891 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc984059ee0 con 0x7fc9940ff9d0 2024-03-24T09:43:03.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:03.567+0000 7fc99c314700 1 -- 172.21.15.22:0/4186955891 --> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fc99419e6e0 con 0x7fc98006cc60 2024-03-24T09:43:03.711 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:03 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:43:03.711 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:03 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:43:03.711 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:03 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:43:03.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:03 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:43:03.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:03 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:43:03.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:03 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:43:04.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:04 smithi022 ceph-mon[29512]: from='client.14272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:43:04.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:04 smithi022 ceph-mon[29512]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:43:04.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:04 smithi155 ceph-mon[37016]: from='client.14272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:43:04.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:04 smithi155 ceph-mon[37016]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:43:07.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:07 smithi155 ceph-mon[37016]: Detected new or changed devices on smithi022 2024-03-24T09:43:07.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:07 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:07.463 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:07 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:07.463 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:07 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:43:07.463 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:07 smithi155 ceph-mon[37016]: Adjusting osd_memory_target on smithi022 to 11845M 2024-03-24T09:43:07.463 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:07 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:07.463 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:07 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:43:07.463 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:07 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:43:07.463 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:07 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:07.463 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:07 smithi155 ceph-mon[37016]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:43:07.463 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:07 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:43:07.689 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:07 smithi022 ceph-mon[29512]: Detected new or changed devices on smithi022 2024-03-24T09:43:07.689 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:07 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:07.689 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:07 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:07.689 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:07 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:43:07.689 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:07 smithi022 ceph-mon[29512]: Adjusting osd_memory_target on smithi022 to 11845M 2024-03-24T09:43:07.689 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:07 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:07.689 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:07 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:43:07.689 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:07 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:43:07.690 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:07 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:07.690 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:07 smithi022 ceph-mon[29512]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:43:07.690 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:07 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:43:08.509 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:08 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/2549412822' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2bc9ee9b-4d76-4df5-bc42-92f243ba6c9d"}]: dispatch 2024-03-24T09:43:08.509 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:08 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/2549412822' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2bc9ee9b-4d76-4df5-bc42-92f243ba6c9d"}]': finished 2024-03-24T09:43:08.509 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:08 smithi022 ceph-mon[29512]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T09:43:08.509 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:08 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T09:43:08.509 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:08 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:08.509 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:08 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:08.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:08 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/2549412822' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2bc9ee9b-4d76-4df5-bc42-92f243ba6c9d"}]: dispatch 2024-03-24T09:43:08.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:08 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/2549412822' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2bc9ee9b-4d76-4df5-bc42-92f243ba6c9d"}]': finished 2024-03-24T09:43:08.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:08 smithi155 ceph-mon[37016]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T09:43:08.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:08 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T09:43:08.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:08 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:08.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:08 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:09.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:09 smithi022 ceph-mon[29512]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:43:09.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:09 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:43:09.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:09 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/165121951' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:43:09.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:09 smithi155 ceph-mon[37016]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:43:09.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:09 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:43:09.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:09 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/165121951' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:43:11.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:11 smithi155 ceph-mon[37016]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:43:11.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:11 smithi022 ceph-mon[29512]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:43:12.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:12 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:12.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:12 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:12.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:12 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:43:12.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:12 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:43:12.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:12 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:12.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:12 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:12.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:12 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:12.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:12 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:43:12.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:12 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:43:12.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:12 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:13.415 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:13 smithi022 ceph-mon[29512]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:43:13.656 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:13 smithi155 ceph-mon[37016]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:43:15.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:15 smithi022 ceph-mon[29512]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:43:15.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:15 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T09:43:15.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:15 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:43:15.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:15 smithi155 ceph-mon[37016]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:43:15.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:15 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T09:43:15.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:15 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:43:16.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:16 smithi022 ceph-mon[29512]: Deploying daemon osd.1 on smithi022 2024-03-24T09:43:16.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:16 smithi155 ceph-mon[37016]: Deploying daemon osd.1 on smithi022 2024-03-24T09:43:17.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:17 smithi022 ceph-mon[29512]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:43:17.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:17 smithi155 ceph-mon[37016]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:43:19.706 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:19 smithi022 ceph-mon[29512]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:43:19.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:19 smithi155 ceph-mon[37016]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:43:21.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:21 smithi155 ceph-mon[37016]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:43:21.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:21 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:21.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:21 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:21.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:21 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:43:21.977 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:21 smithi022 ceph-mon[29512]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:43:21.977 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:21 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:21.977 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:21 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:21.977 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:21 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:43:23.764 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:23 smithi022 ceph-mon[29512]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:43:23.764 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:23 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:23.765 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:23 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:23.765 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:23 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:43:23.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:23 smithi155 ceph-mon[37016]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:43:23.982 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:23 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:23.982 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:23 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:23.982 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:23 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:43:24.471 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:24.471+0000 7fc98b7fe700 1 -- 172.21.15.22:0/4186955891 <== mgr.14215 v2:172.21.15.22:6800/773863266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fc99419e6e0 con 0x7fc98006cc60 2024-03-24T09:43:24.472 INFO:teuthology.orchestra.run.smithi022.stdout:Created osd(s) 1 on host 'smithi022' 2024-03-24T09:43:24.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:24.473+0000 7fc99c314700 1 -- 172.21.15.22:0/4186955891 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fc98006cc60 msgr2=0x7fc98006f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:43:24.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:24.473+0000 7fc99c314700 1 --2- 172.21.15.22:0/4186955891 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fc98006cc60 0x7fc98006f120 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc990005090 tx=0x7fc990005960 comp rx=0 tx=0).stop 2024-03-24T09:43:24.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:24.473+0000 7fc99c314700 1 -- 172.21.15.22:0/4186955891 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc9940ff9d0 msgr2=0x7fc9941038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:43:24.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:24.473+0000 7fc99c314700 1 --2- 172.21.15.22:0/4186955891 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc9940ff9d0 0x7fc9941038c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc98400d900 tx=0x7fc98400dc10 comp rx=0 tx=0).stop 2024-03-24T09:43:24.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:24.473+0000 7fc99c314700 1 -- 172.21.15.22:0/4186955891 shutdown_connections 2024-03-24T09:43:24.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:24.473+0000 7fc99c314700 1 --2- 172.21.15.22:0/4186955891 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fc98006cc60 0x7fc98006f120 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:43:24.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:24.473+0000 7fc99c314700 1 --2- 172.21.15.22:0/4186955891 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc9940ff9d0 0x7fc9941038c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:43:24.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:24.473+0000 7fc99c314700 1 --2- 172.21.15.22:0/4186955891 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc994100bd0 0x7fc994101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:43:24.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:24.473+0000 7fc99c314700 1 -- 172.21.15.22:0/4186955891 >> 172.21.15.22:0/4186955891 conn(0x7fc9940faf70 msgr2=0x7fc994068e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:43:24.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:24.473+0000 7fc99c314700 1 -- 172.21.15.22:0/4186955891 shutdown_connections 2024-03-24T09:43:24.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:24.473+0000 7fc99c314700 1 -- 172.21.15.22:0/4186955891 wait complete. 2024-03-24T09:43:25.271 DEBUG:teuthology.orchestra.run.smithi022:osd.1> sudo journalctl -f -n 0 -u ceph-45356448-e9c2-11ee-95cd-87774f69a715@osd.1.service 2024-03-24T09:43:25.273 INFO:tasks.cephadm:Deploying osd.2 on smithi022 with /dev/vg_nvme/lv_2... 2024-03-24T09:43:25.274 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-24T09:43:25.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:25 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:25.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:25 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:25.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:25 smithi022 ceph-mon[29512]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:43:25.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:25 smithi022 ceph-mon[29512]: from='osd.1 [v2:172.21.15.22:6810/1675089147,v1:172.21.15.22:6811/1675089147]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T09:43:25.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:25 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:25.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:25 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:25.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:25 smithi155 ceph-mon[37016]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:43:25.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:25 smithi155 ceph-mon[37016]: from='osd.1 [v2:172.21.15.22:6810/1675089147,v1:172.21.15.22:6811/1675089147]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T09:43:26.387 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:43:26.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:26 smithi155 ceph-mon[37016]: from='osd.1 [v2:172.21.15.22:6810/1675089147,v1:172.21.15.22:6811/1675089147]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T09:43:26.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:26 smithi155 ceph-mon[37016]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T09:43:26.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:26 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T09:43:26.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:26 smithi155 ceph-mon[37016]: from='osd.1 [v2:172.21.15.22:6810/1675089147,v1:172.21.15.22:6811/1675089147]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2024-03-24T09:43:26.739 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:26 smithi022 ceph-mon[29512]: from='osd.1 [v2:172.21.15.22:6810/1675089147,v1:172.21.15.22:6811/1675089147]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T09:43:26.739 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:26 smithi022 ceph-mon[29512]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T09:43:26.740 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:26 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T09:43:26.740 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:26 smithi022 ceph-mon[29512]: from='osd.1 [v2:172.21.15.22:6810/1675089147,v1:172.21.15.22:6811/1675089147]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2024-03-24T09:43:26.740 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:43:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:43:26.429+0000 7fcd30c78700 -1 osd.1 0 waiting for initial osdmap 2024-03-24T09:43:26.740 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:43:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:43:26.434+0000 7fcd2c26c700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T09:43:27.467 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:27 smithi022 ceph-mon[29512]: from='osd.1 [v2:172.21.15.22:6810/1675089147,v1:172.21.15.22:6811/1675089147]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]': finished 2024-03-24T09:43:27.467 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:27 smithi022 ceph-mon[29512]: osdmap e13: 2 total, 1 up, 2 in 2024-03-24T09:43:27.467 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:27 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T09:43:27.467 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:27 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T09:43:27.467 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:27 smithi022 ceph-mon[29512]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:43:27.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:27 smithi155 ceph-mon[37016]: from='osd.1 [v2:172.21.15.22:6810/1675089147,v1:172.21.15.22:6811/1675089147]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]': finished 2024-03-24T09:43:27.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:27 smithi155 ceph-mon[37016]: osdmap e13: 2 total, 1 up, 2 in 2024-03-24T09:43:27.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:27 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T09:43:27.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:27 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T09:43:27.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:27 smithi155 ceph-mon[37016]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:43:28.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:28 smithi155 ceph-mon[37016]: purged_snaps scrub starts 2024-03-24T09:43:28.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:28 smithi155 ceph-mon[37016]: purged_snaps scrub ok 2024-03-24T09:43:28.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:28 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T09:43:28.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:28 smithi155 ceph-mon[37016]: osd.1 [v2:172.21.15.22:6810/1675089147,v1:172.21.15.22:6811/1675089147] boot 2024-03-24T09:43:28.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:28 smithi155 ceph-mon[37016]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T09:43:28.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:28 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T09:43:28.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:28 smithi022 ceph-mon[29512]: purged_snaps scrub starts 2024-03-24T09:43:28.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:28 smithi022 ceph-mon[29512]: purged_snaps scrub ok 2024-03-24T09:43:28.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:28 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T09:43:28.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:28 smithi022 ceph-mon[29512]: osd.1 [v2:172.21.15.22:6810/1675089147,v1:172.21.15.22:6811/1675089147] boot 2024-03-24T09:43:28.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:28 smithi022 ceph-mon[29512]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T09:43:28.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:28 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T09:43:29.024 INFO:teuthology.orchestra.run.smithi022.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-24T09:43:29.024 INFO:teuthology.orchestra.run.smithi022.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-24T09:43:29.024 INFO:teuthology.orchestra.run.smithi022.stderr: stderr: 10+0 records in 2024-03-24T09:43:29.024 INFO:teuthology.orchestra.run.smithi022.stderr:10+0 records out 2024-03-24T09:43:29.024 INFO:teuthology.orchestra.run.smithi022.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0104098 s, 1.0 GB/s 2024-03-24T09:43:29.024 INFO:teuthology.orchestra.run.smithi022.stderr:--> Zapping successful for: 2024-03-24T09:43:29.653 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:29 smithi022 ceph-mon[29512]: osdmap e15: 2 total, 2 up, 2 in 2024-03-24T09:43:29.654 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:29 smithi022 ceph-mon[29512]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:43:29.654 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:29 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:29.654 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:29 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:29.675 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi022:vg_nvme/lv_2 2024-03-24T09:43:29.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:29 smithi155 ceph-mon[37016]: osdmap e15: 2 total, 2 up, 2 in 2024-03-24T09:43:29.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:29 smithi155 ceph-mon[37016]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:43:29.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:29 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:29.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:29 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:30.756 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:43:31.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:31 smithi022 ceph-mon[29512]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:43:31.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:31 smithi155 ceph-mon[37016]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:43:33.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:33.439+0000 7fcb897b0700 1 -- 172.21.15.22:0/2985481103 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcb840ff9e0 msgr2=0x7fcb840ffe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:43:33.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:33.439+0000 7fcb897b0700 1 --2- 172.21.15.22:0/2985481103 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcb840ff9e0 0x7fcb840ffe00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcb74009910 tx=0x7fcb74009c20 comp rx=0 tx=0).stop 2024-03-24T09:43:33.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:33.439+0000 7fcb897b0700 1 -- 172.21.15.22:0/2985481103 shutdown_connections 2024-03-24T09:43:33.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:33.439+0000 7fcb897b0700 1 --2- 172.21.15.22:0/2985481103 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb84100be0 0x7fcb84101040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:43:33.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:33.439+0000 7fcb897b0700 1 --2- 172.21.15.22:0/2985481103 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcb840ff9e0 0x7fcb840ffe00 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:43:33.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:33.439+0000 7fcb897b0700 1 -- 172.21.15.22:0/2985481103 >> 172.21.15.22:0/2985481103 conn(0x7fcb840faf80 msgr2=0x7fcb840fd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:43:33.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:33.439+0000 7fcb897b0700 1 -- 172.21.15.22:0/2985481103 shutdown_connections 2024-03-24T09:43:33.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:33.439+0000 7fcb897b0700 1 -- 172.21.15.22:0/2985481103 wait complete. 2024-03-24T09:43:33.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:33.440+0000 7fcb897b0700 1 Processor -- start 2024-03-24T09:43:33.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:33.440+0000 7fcb897b0700 1 -- start start 2024-03-24T09:43:33.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:33.440+0000 7fcb897b0700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb840ff9e0 0x7fcb841038d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:43:33.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:33.440+0000 7fcb897b0700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcb84100be0 0x7fcb84101fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:43:33.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:33.440+0000 7fcb897b0700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb84102500 con 0x7fcb84100be0 2024-03-24T09:43:33.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:33.440+0000 7fcb897b0700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb84102640 con 0x7fcb840ff9e0 2024-03-24T09:43:33.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:33.440+0000 7fcb827fc700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcb84100be0 0x7fcb84101fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:43:33.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:33.440+0000 7fcb827fc700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcb84100be0 0x7fcb84101fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:36656/0 (socket says 172.21.15.22:36656) 2024-03-24T09:43:33.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:33.440+0000 7fcb827fc700 1 -- 172.21.15.22:0/3018112925 learned_addr learned my addr 172.21.15.22:0/3018112925 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:43:33.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:33.441+0000 7fcb82ffd700 1 --2- 172.21.15.22:0/3018112925 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb840ff9e0 0x7fcb841038d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:43:33.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:33.441+0000 7fcb827fc700 1 -- 172.21.15.22:0/3018112925 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb840ff9e0 msgr2=0x7fcb841038d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:43:33.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:33.441+0000 7fcb827fc700 1 --2- 172.21.15.22:0/3018112925 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb840ff9e0 0x7fcb841038d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:43:33.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:33.441+0000 7fcb827fc700 1 -- 172.21.15.22:0/3018112925 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb740095f0 con 0x7fcb84100be0 2024-03-24T09:43:33.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:33.441+0000 7fcb827fc700 1 --2- 172.21.15.22:0/3018112925 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcb84100be0 0x7fcb84101fc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcb7800d8d0 tx=0x7fcb7800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:43:33.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:33.441+0000 7fcb6bfff700 1 -- 172.21.15.22:0/3018112925 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb780098a0 con 0x7fcb84100be0 2024-03-24T09:43:33.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:33.441+0000 7fcb6bfff700 1 -- 172.21.15.22:0/3018112925 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcb78010460 con 0x7fcb84100be0 2024-03-24T09:43:33.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:33.441+0000 7fcb897b0700 1 -- 172.21.15.22:0/3018112925 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb84102920 con 0x7fcb84100be0 2024-03-24T09:43:33.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:33.441+0000 7fcb6bfff700 1 -- 172.21.15.22:0/3018112925 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb7800f5f0 con 0x7fcb84100be0 2024-03-24T09:43:33.444 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:33.441+0000 7fcb897b0700 1 -- 172.21.15.22:0/3018112925 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb8419e5d0 con 0x7fcb84100be0 2024-03-24T09:43:33.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:33.442+0000 7fcb897b0700 1 -- 172.21.15.22:0/3018112925 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb8404ea80 con 0x7fcb84100be0 2024-03-24T09:43:33.445 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:33.442+0000 7fcb6bfff700 1 -- 172.21.15.22:0/3018112925 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 18) v1 ==== 90786+0+0 (secure 0 0 0) 0x7fcb78009a00 con 0x7fcb84100be0 2024-03-24T09:43:33.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:33.443+0000 7fcb6bfff700 1 --2- 172.21.15.22:0/3018112925 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fcb6c06cc60 0x7fcb6c06f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:43:33.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:33.443+0000 7fcb6bfff700 1 -- 172.21.15.22:0/3018112925 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2414+0+0 (secure 0 0 0) 0x7fcb7808b360 con 0x7fcb84100be0 2024-03-24T09:43:33.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:33.443+0000 7fcb82ffd700 1 --2- 172.21.15.22:0/3018112925 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fcb6c06cc60 0x7fcb6c06f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:43:33.446 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:33.443+0000 7fcb82ffd700 1 --2- 172.21.15.22:0/3018112925 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fcb6c06cc60 0x7fcb6c06f120 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fcb74000c00 tx=0x7fcb74005960 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:43:33.448 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:33.448+0000 7fcb6bfff700 1 -- 172.21.15.22:0/3018112925 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fcb78059f10 con 0x7fcb84100be0 2024-03-24T09:43:33.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:33 smithi022 ceph-mon[29512]: Detected new or changed devices on smithi022 2024-03-24T09:43:33.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:33 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:33.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:33 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:33.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:33 smithi022 ceph-mon[29512]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:43:33.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:33 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:43:33.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:33 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:43:33.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:33 smithi022 ceph-mon[29512]: Adjusting osd_memory_target on smithi022 to 5922M 2024-03-24T09:43:33.533 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:33 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:33.533 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:33 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:43:33.533 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:33 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:43:33.533 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:33 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:33.533 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:33 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:43:33.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:33.666+0000 7fcb897b0700 1 -- 172.21.15.22:0/3018112925 --> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fcb84069790 con 0x7fcb6c06cc60 2024-03-24T09:43:33.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:33 smithi155 ceph-mon[37016]: Detected new or changed devices on smithi022 2024-03-24T09:43:33.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:33 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:33.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:33 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:33.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:33 smithi155 ceph-mon[37016]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:43:33.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:33 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:43:33.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:33 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:43:33.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:33 smithi155 ceph-mon[37016]: Adjusting osd_memory_target on smithi022 to 5922M 2024-03-24T09:43:33.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:33 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:33.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:33 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:43:33.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:33 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:43:33.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:33 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:33.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:33 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:43:34.658 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:34 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:43:34.658 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:34 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:43:34.659 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:34 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:43:34.659 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:34 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:34.659 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:34 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:34.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:34 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:43:34.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:34 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:43:34.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:34 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:43:34.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:34 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:34.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:34 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:35.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:35 smithi022 ceph-mon[29512]: from='client.14290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:43:35.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:35 smithi022 ceph-mon[29512]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:43:35.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:35 smithi155 ceph-mon[37016]: from='client.14290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi022:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:43:35.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:35 smithi155 ceph-mon[37016]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:43:37.667 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:37 smithi022 ceph-mon[29512]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:43:37.668 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:37 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/3713788085' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e0c3afd-b47b-4de8-ab3c-e25770557c6e"}]: dispatch 2024-03-24T09:43:37.668 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:37 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/3713788085' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e0c3afd-b47b-4de8-ab3c-e25770557c6e"}]': finished 2024-03-24T09:43:37.668 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:37 smithi022 ceph-mon[29512]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T09:43:37.668 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:37 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T09:43:37.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:37 smithi155 ceph-mon[37016]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:43:37.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:37 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/3713788085' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e0c3afd-b47b-4de8-ab3c-e25770557c6e"}]: dispatch 2024-03-24T09:43:37.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:37 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/3713788085' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e0c3afd-b47b-4de8-ab3c-e25770557c6e"}]': finished 2024-03-24T09:43:37.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:37 smithi155 ceph-mon[37016]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T09:43:37.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:37 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T09:43:38.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:38 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/4047258271' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:43:38.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:38 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/4047258271' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:43:39.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:39 smithi022 ceph-mon[29512]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:43:39.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:39 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:43:39.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:39 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:39.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:39 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:39.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:39 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:43:39.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:39 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:43:39.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:39 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:39.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:39 smithi155 ceph-mon[37016]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:43:39.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:39 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:43:39.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:39 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:39.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:39 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:39.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:39 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:43:39.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:39 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:43:39.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:39 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:41.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:41 smithi155 ceph-mon[37016]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:43:41.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:41 smithi022 ceph-mon[29512]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:43:43.723 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:43 smithi022 ceph-mon[29512]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:43:43.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:43 smithi155 ceph-mon[37016]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:43:44.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:44 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T09:43:44.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:44 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:43:44.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:44 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T09:43:44.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:44 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:43:45.745 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:45 smithi022 ceph-mon[29512]: Deploying daemon osd.2 on smithi022 2024-03-24T09:43:45.745 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:45 smithi022 ceph-mon[29512]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:43:45.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:45 smithi155 ceph-mon[37016]: Deploying daemon osd.2 on smithi022 2024-03-24T09:43:45.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:45 smithi155 ceph-mon[37016]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:43:47.655 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:47 smithi022 ceph-mon[29512]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:43:47.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:47 smithi155 ceph-mon[37016]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:43:49.568 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:49 smithi022 ceph-mon[29512]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:43:49.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:49 smithi155 ceph-mon[37016]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:43:51.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:50 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:51.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:50 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:51.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:50 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:43:51.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:50 smithi022 ceph-mon[29512]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:43:51.211 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:50 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:51.211 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:50 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:51.211 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:50 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:43:51.211 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:50 smithi155 ceph-mon[37016]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:43:53.478 INFO:teuthology.orchestra.run.smithi022.stdout:Created osd(s) 2 on host 'smithi022' 2024-03-24T09:43:53.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:53.477+0000 7fcb6bfff700 1 -- 172.21.15.22:0/3018112925 <== mgr.14215 v2:172.21.15.22:6800/773863266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fcb84069790 con 0x7fcb6c06cc60 2024-03-24T09:43:53.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:53.479+0000 7fcb897b0700 1 -- 172.21.15.22:0/3018112925 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fcb6c06cc60 msgr2=0x7fcb6c06f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:43:53.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:53.479+0000 7fcb897b0700 1 --2- 172.21.15.22:0/3018112925 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fcb6c06cc60 0x7fcb6c06f120 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fcb74000c00 tx=0x7fcb74005960 comp rx=0 tx=0).stop 2024-03-24T09:43:53.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:53.479+0000 7fcb897b0700 1 -- 172.21.15.22:0/3018112925 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcb84100be0 msgr2=0x7fcb84101fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:43:53.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:53.479+0000 7fcb897b0700 1 --2- 172.21.15.22:0/3018112925 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcb84100be0 0x7fcb84101fc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcb7800d8d0 tx=0x7fcb7800dbe0 comp rx=0 tx=0).stop 2024-03-24T09:43:53.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:53.479+0000 7fcb897b0700 1 -- 172.21.15.22:0/3018112925 shutdown_connections 2024-03-24T09:43:53.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:53.479+0000 7fcb897b0700 1 --2- 172.21.15.22:0/3018112925 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fcb6c06cc60 0x7fcb6c06f120 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:43:53.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:53.479+0000 7fcb897b0700 1 --2- 172.21.15.22:0/3018112925 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb840ff9e0 0x7fcb841038d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:43:53.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:53.479+0000 7fcb897b0700 1 --2- 172.21.15.22:0/3018112925 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcb84100be0 0x7fcb84101fc0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:43:53.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:53.479+0000 7fcb897b0700 1 -- 172.21.15.22:0/3018112925 >> 172.21.15.22:0/3018112925 conn(0x7fcb840faf80 msgr2=0x7fcb84069080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:43:53.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:53.480+0000 7fcb897b0700 1 -- 172.21.15.22:0/3018112925 shutdown_connections 2024-03-24T09:43:53.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:43:53.480+0000 7fcb897b0700 1 -- 172.21.15.22:0/3018112925 wait complete. 2024-03-24T09:43:53.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:53 smithi022 ceph-mon[29512]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:43:53.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:53 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:53.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:53 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:53.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:53 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:43:53.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:53 smithi155 ceph-mon[37016]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:43:53.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:53 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:53.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:53 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:53.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:53 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:43:54.248 DEBUG:teuthology.orchestra.run.smithi022:osd.2> sudo journalctl -f -n 0 -u ceph-45356448-e9c2-11ee-95cd-87774f69a715@osd.2.service 2024-03-24T09:43:54.252 INFO:tasks.cephadm:Deploying osd.3 on smithi155 with /dev/vg_nvme/lv_4... 2024-03-24T09:43:54.252 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-24T09:43:54.468 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi155/config 2024-03-24T09:43:54.589 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:54 smithi155 ceph-mon[37016]: from='osd.2 [v2:172.21.15.22:6818/1538987315,v1:172.21.15.22:6819/1538987315]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T09:43:54.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:54 smithi022 ceph-mon[29512]: from='osd.2 [v2:172.21.15.22:6818/1538987315,v1:172.21.15.22:6819/1538987315]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T09:43:55.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:55 smithi155 ceph-mon[37016]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:43:55.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:55 smithi155 ceph-mon[37016]: from='osd.2 [v2:172.21.15.22:6818/1538987315,v1:172.21.15.22:6819/1538987315]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T09:43:55.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:55 smithi155 ceph-mon[37016]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T09:43:55.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:55 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T09:43:55.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:55 smithi155 ceph-mon[37016]: from='osd.2 [v2:172.21.15.22:6818/1538987315,v1:172.21.15.22:6819/1538987315]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2024-03-24T09:43:55.833 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:55 smithi022 ceph-mon[29512]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:43:55.833 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:55 smithi022 ceph-mon[29512]: from='osd.2 [v2:172.21.15.22:6818/1538987315,v1:172.21.15.22:6819/1538987315]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T09:43:55.833 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:55 smithi022 ceph-mon[29512]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T09:43:55.834 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:55 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T09:43:55.834 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:55 smithi022 ceph-mon[29512]: from='osd.2 [v2:172.21.15.22:6818/1538987315,v1:172.21.15.22:6819/1538987315]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]: dispatch 2024-03-24T09:43:55.834 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:43:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:43:55.537+0000 7f65a26d3700 -1 osd.2 0 waiting for initial osdmap 2024-03-24T09:43:55.834 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:43:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:43:55.541+0000 7f659ecc9700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T09:43:56.288 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-24T09:43:56.288 INFO:teuthology.orchestra.run.smithi155.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-24T09:43:56.288 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10+0 records in 2024-03-24T09:43:56.288 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records out 2024-03-24T09:43:56.288 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114227 s, 918 MB/s 2024-03-24T09:43:56.288 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping successful for: 2024-03-24T09:43:56.910 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:56 smithi155 ceph-mon[37016]: from='osd.2 [v2:172.21.15.22:6818/1538987315,v1:172.21.15.22:6819/1538987315]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]': finished 2024-03-24T09:43:56.910 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:56 smithi155 ceph-mon[37016]: osdmap e18: 3 total, 2 up, 3 in 2024-03-24T09:43:56.910 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:56 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T09:43:56.910 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:56 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T09:43:56.910 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:56 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:56.910 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:56 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:56.931 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi155:vg_nvme/lv_4 2024-03-24T09:43:56.979 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:56 smithi022 ceph-mon[29512]: from='osd.2 [v2:172.21.15.22:6818/1538987315,v1:172.21.15.22:6819/1538987315]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi022", "root=default"]}]': finished 2024-03-24T09:43:56.979 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:56 smithi022 ceph-mon[29512]: osdmap e18: 3 total, 2 up, 3 in 2024-03-24T09:43:56.980 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:56 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T09:43:56.980 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:56 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T09:43:56.980 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:56 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:56.980 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:56 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:57.137 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi155/config 2024-03-24T09:43:57.906 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:57 smithi155 ceph-mon[37016]: purged_snaps scrub starts 2024-03-24T09:43:57.906 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:57 smithi155 ceph-mon[37016]: purged_snaps scrub ok 2024-03-24T09:43:57.906 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:57 smithi155 ceph-mon[37016]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:43:57.906 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:57 smithi155 ceph-mon[37016]: osd.2 [v2:172.21.15.22:6818/1538987315,v1:172.21.15.22:6819/1538987315] boot 2024-03-24T09:43:57.906 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:57 smithi155 ceph-mon[37016]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T09:43:57.907 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:57 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T09:43:57.907 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:57 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T09:43:57.907 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:57 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T09:43:57.907 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:57 smithi155 ceph-mon[37016]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T09:43:57.907 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:57 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T09:43:57.918 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:57 smithi022 ceph-mon[29512]: purged_snaps scrub starts 2024-03-24T09:43:57.918 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:57 smithi022 ceph-mon[29512]: purged_snaps scrub ok 2024-03-24T09:43:57.918 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:57 smithi022 ceph-mon[29512]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:43:57.919 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:57 smithi022 ceph-mon[29512]: osd.2 [v2:172.21.15.22:6818/1538987315,v1:172.21.15.22:6819/1538987315] boot 2024-03-24T09:43:57.919 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:57 smithi022 ceph-mon[29512]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T09:43:57.919 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:57 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T09:43:57.919 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:57 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T09:43:57.919 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:57 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T09:43:57.919 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:57 smithi022 ceph-mon[29512]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T09:43:57.919 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:57 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T09:43:58.642 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:43:58.636+0000 7fa9dd6dd700 1 -- 172.21.15.155:0/4264152250 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9d80ff9d0 msgr2=0x7fa9d80ffdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:43:58.643 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:43:58.636+0000 7fa9dd6dd700 1 --2- 172.21.15.155:0/4264152250 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9d80ff9d0 0x7fa9d80ffdf0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fa9c8009910 tx=0x7fa9c8009c20 comp rx=0 tx=0).stop 2024-03-24T09:43:58.643 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:43:58.636+0000 7fa9dd6dd700 1 -- 172.21.15.155:0/4264152250 shutdown_connections 2024-03-24T09:43:58.643 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:43:58.636+0000 7fa9dd6dd700 1 --2- 172.21.15.155:0/4264152250 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa9d8100bd0 0x7fa9d8101030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:43:58.643 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:43:58.636+0000 7fa9dd6dd700 1 --2- 172.21.15.155:0/4264152250 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9d80ff9d0 0x7fa9d80ffdf0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:43:58.643 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:43:58.636+0000 7fa9dd6dd700 1 -- 172.21.15.155:0/4264152250 >> 172.21.15.155:0/4264152250 conn(0x7fa9d80faf70 msgr2=0x7fa9d80fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:43:58.643 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:43:58.636+0000 7fa9dd6dd700 1 -- 172.21.15.155:0/4264152250 shutdown_connections 2024-03-24T09:43:58.643 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:43:58.636+0000 7fa9dd6dd700 1 -- 172.21.15.155:0/4264152250 wait complete. 2024-03-24T09:43:58.643 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:43:58.637+0000 7fa9dd6dd700 1 Processor -- start 2024-03-24T09:43:58.644 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:43:58.637+0000 7fa9dd6dd700 1 -- start start 2024-03-24T09:43:58.644 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:43:58.637+0000 7fa9dd6dd700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa9d80ff9d0 0x7fa9d8068f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:43:58.644 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:43:58.637+0000 7fa9dd6dd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9d8100bd0 0x7fa9d819d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:43:58.644 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:43:58.637+0000 7fa9dd6dd700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d8197060 con 0x7fa9d80ff9d0 2024-03-24T09:43:58.644 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:43:58.637+0000 7fa9dd6dd700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d819dbc0 con 0x7fa9d8100bd0 2024-03-24T09:43:58.644 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:43:58.637+0000 7fa9d67fc700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9d8100bd0 0x7fa9d819d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:43:58.644 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:43:58.637+0000 7fa9d67fc700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9d8100bd0 0x7fa9d819d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:52894/0 (socket says 172.21.15.155:52894) 2024-03-24T09:43:58.644 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:43:58.637+0000 7fa9d67fc700 1 -- 172.21.15.155:0/313268482 learned_addr learned my addr 172.21.15.155:0/313268482 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-24T09:43:58.644 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:43:58.637+0000 7fa9d6ffd700 1 --2- 172.21.15.155:0/313268482 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa9d80ff9d0 0x7fa9d8068f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:43:58.645 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:43:58.637+0000 7fa9d67fc700 1 -- 172.21.15.155:0/313268482 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa9d80ff9d0 msgr2=0x7fa9d8068f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:43:58.645 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:43:58.637+0000 7fa9d67fc700 1 --2- 172.21.15.155:0/313268482 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa9d80ff9d0 0x7fa9d8068f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:43:58.645 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:43:58.637+0000 7fa9d67fc700 1 -- 172.21.15.155:0/313268482 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c80095f0 con 0x7fa9d8100bd0 2024-03-24T09:43:58.645 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:43:58.637+0000 7fa9d6ffd700 1 --2- 172.21.15.155:0/313268482 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa9d80ff9d0 0x7fa9d8068f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:43:58.645 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:43:58.637+0000 7fa9d67fc700 1 --2- 172.21.15.155:0/313268482 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9d8100bd0 0x7fa9d819d680 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc00d8d0 tx=0x7fa9cc00dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:43:58.645 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:43:58.638+0000 7fa9bffff700 1 -- 172.21.15.155:0/313268482 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9cc009940 con 0x7fa9d8100bd0 2024-03-24T09:43:58.645 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:43:58.638+0000 7fa9bffff700 1 -- 172.21.15.155:0/313268482 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa9cc010460 con 0x7fa9d8100bd0 2024-03-24T09:43:58.645 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:43:58.638+0000 7fa9dd6dd700 1 -- 172.21.15.155:0/313268482 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9d819ddc0 con 0x7fa9d8100bd0 2024-03-24T09:43:58.645 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:43:58.638+0000 7fa9bffff700 1 -- 172.21.15.155:0/313268482 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9cc00f5f0 con 0x7fa9d8100bd0 2024-03-24T09:43:58.645 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:43:58.638+0000 7fa9dd6dd700 1 -- 172.21.15.155:0/313268482 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9d819e310 con 0x7fa9d8100bd0 2024-03-24T09:43:58.646 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:43:58.639+0000 7fa9dd6dd700 1 -- 172.21.15.155:0/313268482 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9d818f550 con 0x7fa9d8100bd0 2024-03-24T09:43:58.646 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:43:58.639+0000 7fa9bffff700 1 -- 172.21.15.155:0/313268482 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 18) v1 ==== 90786+0+0 (secure 0 0 0) 0x7fa9cc00f820 con 0x7fa9d8100bd0 2024-03-24T09:43:58.646 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:43:58.639+0000 7fa9bffff700 1 --2- 172.21.15.155:0/313268482 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fa9c006cc60 0x7fa9c006f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:43:58.646 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:43:58.639+0000 7fa9bffff700 1 -- 172.21.15.155:0/313268482 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3232+0+0 (secure 0 0 0) 0x7fa9cc08b7e0 con 0x7fa9d8100bd0 2024-03-24T09:43:58.646 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:43:58.640+0000 7fa9d6ffd700 1 --2- 172.21.15.155:0/313268482 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fa9c006cc60 0x7fa9c006f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:43:58.646 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:43:58.640+0000 7fa9d6ffd700 1 --2- 172.21.15.155:0/313268482 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fa9c006cc60 0x7fa9c006f120 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa9c8006010 tx=0x7fa9c8005a90 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:43:58.647 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:43:58.644+0000 7fa9bffff700 1 -- 172.21.15.155:0/313268482 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa9cc05a0d0 con 0x7fa9d8100bd0 2024-03-24T09:43:58.852 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:43:58.851+0000 7fa9dd6dd700 1 -- 172.21.15.155:0/313268482 --> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fa9d80611f0 con 0x7fa9c006cc60 2024-03-24T09:43:59.165 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:43:58 smithi022 sudo[68953]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T09:43:59.165 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:43:58 smithi022 sudo[68953]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T09:43:59.165 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:43:58 smithi022 sudo[68953]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T09:43:59.165 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:43:58 smithi022 sudo[68953]: pam_unix(sudo:session): session closed for user root 2024-03-24T09:43:59.448 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:43:59 smithi022 sudo[68959]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T09:43:59.448 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:43:59 smithi022 sudo[68959]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T09:43:59.448 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:43:59 smithi022 sudo[68959]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T09:43:59.449 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:43:59 smithi022 sudo[68959]: pam_unix(sudo:session): session closed for user root 2024-03-24T09:43:59.604 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:59 smithi155 ceph-mon[37016]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:43:59.604 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:59 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T09:43:59.605 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:59 smithi155 ceph-mon[37016]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T09:43:59.605 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:59 smithi155 ceph-mon[37016]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T09:43:59.605 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:59 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:43:59.605 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:59 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:43:59.605 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:59 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:43:59.605 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:59 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:59.605 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:59 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:59.605 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:59 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:59.605 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:59 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:43:59.605 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:59 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:43:59.606 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:59 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:43:59.606 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:59 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:59.606 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:59 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:43:59.606 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:59 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:43:59.606 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:43:59 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:59.775 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:43:59 smithi022 sudo[68962]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T09:43:59.775 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:43:59 smithi022 sudo[68962]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T09:43:59.775 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:43:59 smithi022 sudo[68962]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T09:43:59.775 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:43:59 smithi022 sudo[68962]: pam_unix(sudo:session): session closed for user root 2024-03-24T09:43:59.776 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:59 smithi022 ceph-mon[29512]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:43:59.776 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:59 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T09:43:59.776 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:59 smithi022 ceph-mon[29512]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T09:43:59.776 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:59 smithi022 ceph-mon[29512]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T09:43:59.776 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:59 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:43:59.776 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:59 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:43:59.776 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:59 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:43:59.776 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:59 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:59.777 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:59 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:59.777 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:59 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:59.777 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:59 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:43:59.777 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:59 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:43:59.777 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:59 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:43:59.777 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:59 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:43:59.777 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:59 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:43:59.777 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:59 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:43:59.777 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:43:59 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:00.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:43:59 smithi022 sudo[69063]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T09:44:00.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:43:59 smithi022 sudo[69063]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T09:44:00.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:43:59 smithi022 sudo[69063]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T09:44:00.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:43:59 smithi022 sudo[69063]: pam_unix(sudo:session): session closed for user root 2024-03-24T09:44:00.246 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:00.245+0000 7fa9bffff700 1 -- 172.21.15.155:0/313268482 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7fa9cc08dce0 con 0x7fa9d8100bd0 2024-03-24T09:44:00.309 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:44:00 smithi022 sudo[69124]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T09:44:00.309 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:44:00 smithi022 sudo[69124]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T09:44:00.310 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:44:00 smithi022 sudo[69124]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T09:44:00.310 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:44:00 smithi022 sudo[69124]: pam_unix(sudo:session): session closed for user root 2024-03-24T09:44:00.568 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:44:00 smithi022 sudo[69159]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T09:44:00.568 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:44:00 smithi022 sudo[69159]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T09:44:00.569 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:44:00 smithi022 sudo[69159]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T09:44:00.569 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:44:00 smithi022 sudo[69159]: pam_unix(sudo:session): session closed for user root 2024-03-24T09:44:00.847 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:00 smithi022 sudo[69176]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T09:44:00.847 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:00 smithi022 sudo[69176]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T09:44:00.847 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:00 smithi022 sudo[69176]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T09:44:00.847 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:00 smithi022 sudo[69176]: pam_unix(sudo:session): session closed for user root 2024-03-24T09:44:00.847 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:00 smithi022 sudo[69232]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T09:44:01.103 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:00 smithi155 ceph-mon[37016]: from='client.24131 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:44:01.104 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:00 smithi155 ceph-mon[37016]: Detected new or changed devices on smithi022 2024-03-24T09:44:01.104 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:00 smithi155 ceph-mon[37016]: Adjusting osd_memory_target on smithi022 to 3948M 2024-03-24T09:44:01.104 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:00 smithi155 ceph-mon[37016]: osdmap e22: 3 total, 3 up, 3 in 2024-03-24T09:44:01.104 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:00 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:44:01.104 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:00 smithi155 ceph-mon[37016]: mgrmap e19: smithi022.emgmxz(active, since 2m), standbys: smithi155.itwzew 2024-03-24T09:44:01.104 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:00 smithi155 ceph-mon[37016]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T09:44:01.121 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:00 smithi022 sudo[69232]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T09:44:01.121 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:00 smithi022 sudo[69232]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T09:44:01.121 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:00 smithi022 sudo[69232]: pam_unix(sudo:session): session closed for user root 2024-03-24T09:44:01.121 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:00 smithi022 ceph-mon[29512]: from='client.24131 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:44:01.121 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:00 smithi022 ceph-mon[29512]: Detected new or changed devices on smithi022 2024-03-24T09:44:01.121 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:00 smithi022 ceph-mon[29512]: Adjusting osd_memory_target on smithi022 to 3948M 2024-03-24T09:44:01.122 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:00 smithi022 ceph-mon[29512]: osdmap e22: 3 total, 3 up, 3 in 2024-03-24T09:44:01.122 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:00 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:44:01.122 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:00 smithi022 ceph-mon[29512]: mgrmap e19: smithi022.emgmxz(active, since 2m), standbys: smithi155.itwzew 2024-03-24T09:44:01.122 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:00 smithi022 ceph-mon[29512]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T09:44:01.359 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:01 smithi155 sudo[40999]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T09:44:01.359 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:01 smithi155 sudo[40999]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T09:44:01.359 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:01 smithi155 sudo[40999]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T09:44:01.359 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:01 smithi155 sudo[40999]: pam_unix(sudo:session): session closed for user root 2024-03-24T09:44:01.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:01 smithi155 sudo[41013]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T09:44:01.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:01 smithi155 sudo[41013]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T09:44:01.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:01 smithi155 sudo[41013]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T09:44:01.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:01 smithi155 sudo[41013]: pam_unix(sudo:session): session closed for user root 2024-03-24T09:44:02.211 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:01 smithi155 ceph-mon[37016]: pgmap v81: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:44:02.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:01 smithi155 ceph-mon[37016]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T09:44:02.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:01 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mon metadata", "id": "smithi022"}]: dispatch 2024-03-24T09:44:02.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:01 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mon metadata", "id": "smithi155"}]: dispatch 2024-03-24T09:44:02.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:01 smithi155 ceph-mon[37016]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T09:44:02.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:01 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mon metadata", "id": "smithi022"}]: dispatch 2024-03-24T09:44:02.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:01 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mon metadata", "id": "smithi155"}]: dispatch 2024-03-24T09:44:02.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:01 smithi155 ceph-mon[37016]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T09:44:02.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:01 smithi155 ceph-mon[37016]: Cluster is now healthy 2024-03-24T09:44:02.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:01 smithi155 ceph-mon[37016]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T09:44:02.213 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:01 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:02.213 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:01 smithi155 ceph-mon[37016]: from='client.? 172.21.15.155:0/3365385115' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56f51449-b4b8-4874-9469-497ce984a5f6"}]: dispatch 2024-03-24T09:44:02.213 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:01 smithi155 ceph-mon[37016]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56f51449-b4b8-4874-9469-497ce984a5f6"}]: dispatch 2024-03-24T09:44:02.213 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:01 smithi155 ceph-mon[37016]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "56f51449-b4b8-4874-9469-497ce984a5f6"}]': finished 2024-03-24T09:44:02.213 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:01 smithi155 ceph-mon[37016]: osdmap e23: 4 total, 3 up, 4 in 2024-03-24T09:44:02.213 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:01 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T09:44:02.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:01 smithi022 ceph-mon[29512]: pgmap v81: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:44:02.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:01 smithi022 ceph-mon[29512]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T09:44:02.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:01 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mon metadata", "id": "smithi022"}]: dispatch 2024-03-24T09:44:02.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:01 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mon metadata", "id": "smithi155"}]: dispatch 2024-03-24T09:44:02.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:01 smithi022 ceph-mon[29512]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T09:44:02.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:01 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mon metadata", "id": "smithi022"}]: dispatch 2024-03-24T09:44:02.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:01 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mon metadata", "id": "smithi155"}]: dispatch 2024-03-24T09:44:02.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:01 smithi022 ceph-mon[29512]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T09:44:02.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:01 smithi022 ceph-mon[29512]: Cluster is now healthy 2024-03-24T09:44:02.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:01 smithi022 ceph-mon[29512]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T09:44:02.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:01 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:02.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:01 smithi022 ceph-mon[29512]: from='client.? 172.21.15.155:0/3365385115' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56f51449-b4b8-4874-9469-497ce984a5f6"}]: dispatch 2024-03-24T09:44:02.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:01 smithi022 ceph-mon[29512]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56f51449-b4b8-4874-9469-497ce984a5f6"}]: dispatch 2024-03-24T09:44:02.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:01 smithi022 ceph-mon[29512]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "56f51449-b4b8-4874-9469-497ce984a5f6"}]': finished 2024-03-24T09:44:02.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:01 smithi022 ceph-mon[29512]: osdmap e23: 4 total, 3 up, 4 in 2024-03-24T09:44:02.284 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:01 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T09:44:03.211 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:02 smithi155 ceph-mon[37016]: from='client.? 172.21.15.155:0/2072116311' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:44:03.211 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:02 smithi155 ceph-mon[37016]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:44:03.213 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:02 smithi022 ceph-mon[29512]: from='client.? 172.21.15.155:0/2072116311' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:44:03.213 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:02 smithi022 ceph-mon[29512]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:44:04.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:04 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:04.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:04 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:04.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:04 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:44:04.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:04 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:44:04.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:04 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:04.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:04 smithi155 ceph-mon[37016]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:44:05.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:04 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:05.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:04 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:05.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:04 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:44:05.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:04 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:44:05.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:04 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:05.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:04 smithi022 ceph-mon[29512]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:44:07.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:07 smithi022 ceph-mon[29512]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:44:07.840 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:07 smithi155 ceph-mon[37016]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:44:08.702 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:08 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:44:09.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:08 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:44:09.854 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:09 smithi022 ceph-mon[29512]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:44:09.854 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:09 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T09:44:09.854 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:09 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:44:09.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:09 smithi155 ceph-mon[37016]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:44:09.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:09 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T09:44:09.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:09 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:44:10.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:10 smithi155 ceph-mon[37016]: Deploying daemon osd.3 on smithi155 2024-03-24T09:44:11.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:10 smithi022 ceph-mon[29512]: Deploying daemon osd.3 on smithi155 2024-03-24T09:44:11.806 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:11 smithi155 ceph-mon[37016]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:44:12.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:11 smithi022 ceph-mon[29512]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:44:13.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:13 smithi155 ceph-mon[37016]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:44:14.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:13 smithi022 ceph-mon[29512]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:44:15.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:15 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:15.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:15 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:44:15.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:15 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:15.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:15 smithi022 ceph-mon[29512]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:44:15.438 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:15 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:15.438 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:15 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:44:15.438 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:15 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:15.438 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:15 smithi155 ceph-mon[37016]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:44:17.566 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:17.565+0000 7fa9bffff700 1 -- 172.21.15.155:0/313268482 <== mgr.14215 v2:172.21.15.22:6800/773863266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fa9d80611f0 con 0x7fa9c006cc60 2024-03-24T09:44:17.566 INFO:teuthology.orchestra.run.smithi155.stdout:Created osd(s) 3 on host 'smithi155' 2024-03-24T09:44:17.567 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:17.567+0000 7fa9dd6dd700 1 -- 172.21.15.155:0/313268482 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fa9c006cc60 msgr2=0x7fa9c006f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:44:17.567 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:17.567+0000 7fa9dd6dd700 1 --2- 172.21.15.155:0/313268482 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fa9c006cc60 0x7fa9c006f120 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa9c8006010 tx=0x7fa9c8005a90 comp rx=0 tx=0).stop 2024-03-24T09:44:17.567 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:17.567+0000 7fa9dd6dd700 1 -- 172.21.15.155:0/313268482 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9d8100bd0 msgr2=0x7fa9d819d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:44:17.568 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:17.567+0000 7fa9dd6dd700 1 --2- 172.21.15.155:0/313268482 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9d8100bd0 0x7fa9d819d680 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc00d8d0 tx=0x7fa9cc00dc90 comp rx=0 tx=0).stop 2024-03-24T09:44:17.568 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:17.567+0000 7fa9dd6dd700 1 -- 172.21.15.155:0/313268482 shutdown_connections 2024-03-24T09:44:17.568 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:17.567+0000 7fa9dd6dd700 1 --2- 172.21.15.155:0/313268482 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fa9c006cc60 0x7fa9c006f120 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:44:17.568 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:17.567+0000 7fa9dd6dd700 1 --2- 172.21.15.155:0/313268482 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa9d80ff9d0 0x7fa9d8068f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:44:17.568 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:17.567+0000 7fa9dd6dd700 1 --2- 172.21.15.155:0/313268482 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9d8100bd0 0x7fa9d819d680 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:44:17.568 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:17.567+0000 7fa9dd6dd700 1 -- 172.21.15.155:0/313268482 >> 172.21.15.155:0/313268482 conn(0x7fa9d80faf70 msgr2=0x7fa9d8104220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:44:17.568 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:17.567+0000 7fa9dd6dd700 1 -- 172.21.15.155:0/313268482 shutdown_connections 2024-03-24T09:44:17.568 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:17.567+0000 7fa9dd6dd700 1 -- 172.21.15.155:0/313268482 wait complete. 2024-03-24T09:44:17.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:17 smithi022 ceph-mon[29512]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:44:17.907 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:17 smithi155 ceph-mon[37016]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:44:18.432 DEBUG:teuthology.orchestra.run.smithi155:osd.3> sudo journalctl -f -n 0 -u ceph-45356448-e9c2-11ee-95cd-87774f69a715@osd.3.service 2024-03-24T09:44:18.434 INFO:tasks.cephadm:Deploying osd.4 on smithi155 with /dev/vg_nvme/lv_3... 2024-03-24T09:44:18.435 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-24T09:44:18.682 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi155/config 2024-03-24T09:44:18.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:18 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:18.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:18 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:18.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:18 smithi155 ceph-mon[37016]: from='osd.3 [v2:172.21.15.155:6800/2405415706,v1:172.21.15.155:6801/2405415706]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T09:44:18.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:18 smithi155 ceph-mon[37016]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T09:44:19.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:18 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:19.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:18 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:19.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:18 smithi022 ceph-mon[29512]: from='osd.3 [v2:172.21.15.155:6800/2405415706,v1:172.21.15.155:6801/2405415706]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T09:44:19.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:18 smithi022 ceph-mon[29512]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T09:44:19.686 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:19 smithi155 ceph-mon[37016]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:44:19.686 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:19 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:19.686 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:19 smithi155 ceph-mon[37016]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T09:44:19.686 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:19 smithi155 ceph-mon[37016]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T09:44:19.686 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:19 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T09:44:19.686 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:19 smithi155 ceph-mon[37016]: from='osd.3 [v2:172.21.15.155:6800/2405415706,v1:172.21.15.155:6801/2405415706]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-24T09:44:19.686 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:19 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:19.686 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:19 smithi155 ceph-mon[37016]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-24T09:44:19.687 INFO:journalctl@ceph.osd.3.smithi155.stdout:Mar 24 09:44:19 smithi155 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-3[43125]: 2024-03-24T09:44:19.575+0000 7f00e6959700 -1 osd.3 0 waiting for initial osdmap 2024-03-24T09:44:19.687 INFO:journalctl@ceph.osd.3.smithi155.stdout:Mar 24 09:44:19 smithi155 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-3[43125]: 2024-03-24T09:44:19.581+0000 7f00e0f4b700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T09:44:19.854 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:19 smithi022 ceph-mon[29512]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:44:19.854 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:19 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:19.854 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:19 smithi022 ceph-mon[29512]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T09:44:19.854 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:19 smithi022 ceph-mon[29512]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T09:44:19.855 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:19 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T09:44:19.855 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:19 smithi022 ceph-mon[29512]: from='osd.3 [v2:172.21.15.155:6800/2405415706,v1:172.21.15.155:6801/2405415706]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-24T09:44:19.855 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:19 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:19.855 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:19 smithi022 ceph-mon[29512]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-24T09:44:20.867 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:20 smithi155 ceph-mon[37016]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-03-24T09:44:20.867 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:20 smithi155 ceph-mon[37016]: osdmap e25: 4 total, 3 up, 4 in 2024-03-24T09:44:20.867 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:20 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T09:44:20.867 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:20 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T09:44:20.867 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:20 smithi155 ceph-mon[37016]: osd.3 [v2:172.21.15.155:6800/2405415706,v1:172.21.15.155:6801/2405415706] boot 2024-03-24T09:44:20.867 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:20 smithi155 ceph-mon[37016]: osdmap e26: 4 total, 4 up, 4 in 2024-03-24T09:44:20.868 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:20 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T09:44:20.868 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:20 smithi155 ceph-mon[37016]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:44:21.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:20 smithi022 ceph-mon[29512]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-03-24T09:44:21.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:20 smithi022 ceph-mon[29512]: osdmap e25: 4 total, 3 up, 4 in 2024-03-24T09:44:21.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:20 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T09:44:21.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:20 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T09:44:21.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:20 smithi022 ceph-mon[29512]: osd.3 [v2:172.21.15.155:6800/2405415706,v1:172.21.15.155:6801/2405415706] boot 2024-03-24T09:44:21.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:20 smithi022 ceph-mon[29512]: osdmap e26: 4 total, 4 up, 4 in 2024-03-24T09:44:21.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:20 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T09:44:21.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:20 smithi022 ceph-mon[29512]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:44:21.668 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-24T09:44:21.668 INFO:teuthology.orchestra.run.smithi155.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-24T09:44:21.668 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10+0 records in 2024-03-24T09:44:21.668 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records out 2024-03-24T09:44:21.669 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124023 s, 845 MB/s 2024-03-24T09:44:21.669 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping successful for: 2024-03-24T09:44:21.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:21 smithi155 ceph-mon[37016]: purged_snaps scrub starts 2024-03-24T09:44:21.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:21 smithi155 ceph-mon[37016]: purged_snaps scrub ok 2024-03-24T09:44:21.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:21 smithi155 ceph-mon[37016]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T09:44:22.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:21 smithi022 ceph-mon[29512]: purged_snaps scrub starts 2024-03-24T09:44:22.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:21 smithi022 ceph-mon[29512]: purged_snaps scrub ok 2024-03-24T09:44:22.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:21 smithi022 ceph-mon[29512]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T09:44:22.432 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi155:vg_nvme/lv_3 2024-03-24T09:44:23.179 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi155/config 2024-03-24T09:44:23.347 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:23 smithi155 ceph-mon[37016]: osdmap e28: 4 total, 4 up, 4 in 2024-03-24T09:44:23.347 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:23 smithi155 ceph-mon[37016]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T09:44:23.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:23 smithi022 ceph-mon[29512]: osdmap e28: 4 total, 4 up, 4 in 2024-03-24T09:44:23.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:23 smithi022 ceph-mon[29512]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T09:44:24.351 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:24 smithi155 ceph-mon[37016]: Detected new or changed devices on smithi155 2024-03-24T09:44:24.351 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:24 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:24.351 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:24 smithi155 ceph-mon[37016]: osdmap e29: 4 total, 4 up, 4 in 2024-03-24T09:44:24.351 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:24 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:24.352 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:24 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:44:24.352 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:24 smithi155 ceph-mon[37016]: Adjusting osd_memory_target on smithi155 to 14912M 2024-03-24T09:44:24.352 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:24 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:24.352 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:24 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:44:24.352 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:24 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:44:24.352 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:24 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:24.352 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:24 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:44:24.352 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:24 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:44:24.352 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:24 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:44:24.353 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:24 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:24.353 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:24 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:44:24.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:24 smithi022 ceph-mon[29512]: Detected new or changed devices on smithi155 2024-03-24T09:44:24.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:24 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:24.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:24 smithi022 ceph-mon[29512]: osdmap e29: 4 total, 4 up, 4 in 2024-03-24T09:44:24.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:24 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:24.533 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:24 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:44:24.533 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:24 smithi022 ceph-mon[29512]: Adjusting osd_memory_target on smithi155 to 14912M 2024-03-24T09:44:24.533 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:24 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:24.533 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:24 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:44:24.533 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:24 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:44:24.533 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:24 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:24.533 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:24 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:44:24.533 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:24 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:44:24.533 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:24 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:44:24.533 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:24 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:24.534 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:24 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:44:24.574 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:24.574+0000 7fb41fc93700 1 -- 172.21.15.155:0/1582871891 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4180ff9d0 msgr2=0x7fb4180ffdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:44:24.574 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:24.574+0000 7fb41fc93700 1 --2- 172.21.15.155:0/1582871891 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4180ff9d0 0x7fb4180ffdf0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb408009910 tx=0x7fb408009c20 comp rx=0 tx=0).stop 2024-03-24T09:44:24.574 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:24.574+0000 7fb41fc93700 1 -- 172.21.15.155:0/1582871891 shutdown_connections 2024-03-24T09:44:24.574 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:24.574+0000 7fb41fc93700 1 --2- 172.21.15.155:0/1582871891 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb418100bd0 0x7fb418101030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:44:24.575 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:24.574+0000 7fb41fc93700 1 --2- 172.21.15.155:0/1582871891 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4180ff9d0 0x7fb4180ffdf0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:44:24.575 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:24.574+0000 7fb41fc93700 1 -- 172.21.15.155:0/1582871891 >> 172.21.15.155:0/1582871891 conn(0x7fb4180faf70 msgr2=0x7fb4180fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:44:24.575 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:24.574+0000 7fb41fc93700 1 -- 172.21.15.155:0/1582871891 shutdown_connections 2024-03-24T09:44:24.575 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:24.574+0000 7fb41fc93700 1 -- 172.21.15.155:0/1582871891 wait complete. 2024-03-24T09:44:24.575 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:24.575+0000 7fb41fc93700 1 Processor -- start 2024-03-24T09:44:24.576 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:24.575+0000 7fb41fc93700 1 -- start start 2024-03-24T09:44:24.576 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:24.575+0000 7fb41fc93700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4180ff9d0 0x7fb418195420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:44:24.578 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:24.575+0000 7fb41fc93700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb418100bd0 0x7fb418195960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:44:24.578 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:24.575+0000 7fb41fc93700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb418195f80 con 0x7fb418100bd0 2024-03-24T09:44:24.578 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:24.575+0000 7fb41fc93700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4181960c0 con 0x7fb4180ff9d0 2024-03-24T09:44:24.578 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:24.575+0000 7fb41da2f700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4180ff9d0 0x7fb418195420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:44:24.579 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:24.575+0000 7fb41da2f700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4180ff9d0 0x7fb418195420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:57092/0 (socket says 172.21.15.155:57092) 2024-03-24T09:44:24.579 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:24.575+0000 7fb41da2f700 1 -- 172.21.15.155:0/938259656 learned_addr learned my addr 172.21.15.155:0/938259656 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-24T09:44:24.579 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:24.575+0000 7fb41d22e700 1 --2- 172.21.15.155:0/938259656 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb418100bd0 0x7fb418195960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:44:24.579 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:24.575+0000 7fb41da2f700 1 -- 172.21.15.155:0/938259656 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb418100bd0 msgr2=0x7fb418195960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:44:24.579 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:24.575+0000 7fb41da2f700 1 --2- 172.21.15.155:0/938259656 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb418100bd0 0x7fb418195960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:44:24.579 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:24.575+0000 7fb41da2f700 1 -- 172.21.15.155:0/938259656 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4080095f0 con 0x7fb4180ff9d0 2024-03-24T09:44:24.579 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:24.576+0000 7fb41da2f700 1 --2- 172.21.15.155:0/938259656 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4180ff9d0 0x7fb418195420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb4080059d0 tx=0x7fb40800bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:44:24.579 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:24.576+0000 7fb40effd700 1 -- 172.21.15.155:0/938259656 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb40801d070 con 0x7fb4180ff9d0 2024-03-24T09:44:24.579 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:24.576+0000 7fb40effd700 1 -- 172.21.15.155:0/938259656 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb408005dc0 con 0x7fb4180ff9d0 2024-03-24T09:44:24.579 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:24.576+0000 7fb41fc93700 1 -- 172.21.15.155:0/938259656 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb41819aa70 con 0x7fb4180ff9d0 2024-03-24T09:44:24.579 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:24.576+0000 7fb40effd700 1 -- 172.21.15.155:0/938259656 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb408021410 con 0x7fb4180ff9d0 2024-03-24T09:44:24.579 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:24.576+0000 7fb41fc93700 1 -- 172.21.15.155:0/938259656 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb41819afe0 con 0x7fb4180ff9d0 2024-03-24T09:44:24.580 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:24.577+0000 7fb41fc93700 1 -- 172.21.15.155:0/938259656 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb41818f550 con 0x7fb4180ff9d0 2024-03-24T09:44:24.580 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:24.577+0000 7fb40effd700 1 -- 172.21.15.155:0/938259656 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7fb40800f460 con 0x7fb4180ff9d0 2024-03-24T09:44:24.580 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:24.578+0000 7fb40effd700 1 --2- 172.21.15.155:0/938259656 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fb40406cce0 0x7fb40406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:44:24.580 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:24.578+0000 7fb40effd700 1 -- 172.21.15.155:0/938259656 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 3774+0+0 (secure 0 0 0) 0x7fb40808cdb0 con 0x7fb4180ff9d0 2024-03-24T09:44:24.580 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:24.578+0000 7fb41d22e700 1 --2- 172.21.15.155:0/938259656 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fb40406cce0 0x7fb40406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:44:24.580 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:24.578+0000 7fb41d22e700 1 --2- 172.21.15.155:0/938259656 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fb40406cce0 0x7fb40406f1a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb414005950 tx=0x7fb41400b410 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:44:24.582 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:24.582+0000 7fb40effd700 1 -- 172.21.15.155:0/938259656 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb40805b2b0 con 0x7fb4180ff9d0 2024-03-24T09:44:24.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:24.784+0000 7fb41fc93700 1 -- 172.21.15.155:0/938259656 --> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fb4180611f0 con 0x7fb40406cce0 2024-03-24T09:44:25.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:25 smithi155 ceph-mon[37016]: osdmap e30: 4 total, 4 up, 4 in 2024-03-24T09:44:25.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:25 smithi155 ceph-mon[37016]: pgmap v101: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T09:44:25.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:25 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:44:25.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:25 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:44:25.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:25 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:44:25.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:25 smithi022 ceph-mon[29512]: osdmap e30: 4 total, 4 up, 4 in 2024-03-24T09:44:25.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:25 smithi022 ceph-mon[29512]: pgmap v101: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T09:44:25.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:25 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:44:25.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:25 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:44:25.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:25 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:44:26.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:26 smithi022 ceph-mon[29512]: from='client.24155 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:44:26.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:26 smithi155 ceph-mon[37016]: from='client.24155 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:44:27.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:27 smithi155 ceph-mon[37016]: pgmap v102: 1 pgs: 1 peering; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-03-24T09:44:27.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:27 smithi022 ceph-mon[29512]: pgmap v102: 1 pgs: 1 peering; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-03-24T09:44:28.226 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:28 smithi155 ceph-mon[37016]: from='client.? 172.21.15.155:0/3426146205' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1af86297-1514-495a-8d74-6d0237f82316"}]: dispatch 2024-03-24T09:44:28.227 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:28 smithi155 ceph-mon[37016]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1af86297-1514-495a-8d74-6d0237f82316"}]: dispatch 2024-03-24T09:44:28.227 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:28 smithi155 ceph-mon[37016]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1af86297-1514-495a-8d74-6d0237f82316"}]': finished 2024-03-24T09:44:28.227 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:28 smithi155 ceph-mon[37016]: osdmap e31: 5 total, 4 up, 5 in 2024-03-24T09:44:28.227 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:28 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T09:44:28.227 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:28 smithi155 ceph-mon[37016]: from='client.? 172.21.15.155:0/3762539681' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:44:28.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:28 smithi022 ceph-mon[29512]: from='client.? 172.21.15.155:0/3426146205' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1af86297-1514-495a-8d74-6d0237f82316"}]: dispatch 2024-03-24T09:44:28.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:28 smithi022 ceph-mon[29512]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1af86297-1514-495a-8d74-6d0237f82316"}]: dispatch 2024-03-24T09:44:28.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:28 smithi022 ceph-mon[29512]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1af86297-1514-495a-8d74-6d0237f82316"}]': finished 2024-03-24T09:44:28.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:28 smithi022 ceph-mon[29512]: osdmap e31: 5 total, 4 up, 5 in 2024-03-24T09:44:28.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:28 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T09:44:28.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:28 smithi022 ceph-mon[29512]: from='client.? 172.21.15.155:0/3762539681' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:44:29.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:29 smithi155 ceph-mon[37016]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T09:44:29.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:29 smithi022 ceph-mon[29512]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T09:44:31.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:31 smithi155 ceph-mon[37016]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-24T09:44:31.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:31 smithi022 ceph-mon[29512]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-24T09:44:33.870 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:33 smithi155 ceph-mon[37016]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T09:44:34.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:33 smithi022 ceph-mon[29512]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T09:44:35.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:35 smithi155 ceph-mon[37016]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T09:44:35.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:35 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T09:44:35.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:35 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:44:35.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:35 smithi155 ceph-mon[37016]: Deploying daemon osd.4 on smithi155 2024-03-24T09:44:36.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:35 smithi022 ceph-mon[29512]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T09:44:36.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:35 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T09:44:36.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:35 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:44:36.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:35 smithi022 ceph-mon[29512]: Deploying daemon osd.4 on smithi155 2024-03-24T09:44:37.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:37 smithi155 ceph-mon[37016]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T09:44:38.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:37 smithi022 ceph-mon[29512]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T09:44:38.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:38 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:44:39.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:38 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:44:39.854 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:39 smithi022 ceph-mon[29512]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T09:44:39.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:39 smithi155 ceph-mon[37016]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T09:44:41.169 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:40 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:41.169 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:40 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:41.170 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:40 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:44:41.170 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:40 smithi155 ceph-mon[37016]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T09:44:41.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:40 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:41.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:40 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:41.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:40 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:44:41.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:40 smithi022 ceph-mon[29512]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T09:44:43.399 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:43.398+0000 7fb40effd700 1 -- 172.21.15.155:0/938259656 <== mgr.14215 v2:172.21.15.22:6800/773863266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb4180611f0 con 0x7fb40406cce0 2024-03-24T09:44:43.400 INFO:teuthology.orchestra.run.smithi155.stdout:Created osd(s) 4 on host 'smithi155' 2024-03-24T09:44:43.401 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:43.400+0000 7fb41fc93700 1 -- 172.21.15.155:0/938259656 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fb40406cce0 msgr2=0x7fb40406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:44:43.402 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:43.400+0000 7fb41fc93700 1 --2- 172.21.15.155:0/938259656 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fb40406cce0 0x7fb40406f1a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb414005950 tx=0x7fb41400b410 comp rx=0 tx=0).stop 2024-03-24T09:44:43.402 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:43.400+0000 7fb41fc93700 1 -- 172.21.15.155:0/938259656 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4180ff9d0 msgr2=0x7fb418195420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:44:43.402 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:43.400+0000 7fb41fc93700 1 --2- 172.21.15.155:0/938259656 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4180ff9d0 0x7fb418195420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb4080059d0 tx=0x7fb40800bd70 comp rx=0 tx=0).stop 2024-03-24T09:44:43.402 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:43.400+0000 7fb41fc93700 1 -- 172.21.15.155:0/938259656 shutdown_connections 2024-03-24T09:44:43.402 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:43.400+0000 7fb41fc93700 1 --2- 172.21.15.155:0/938259656 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fb40406cce0 0x7fb40406f1a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:44:43.402 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:43.400+0000 7fb41fc93700 1 --2- 172.21.15.155:0/938259656 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4180ff9d0 0x7fb418195420 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:44:43.402 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:43.400+0000 7fb41fc93700 1 --2- 172.21.15.155:0/938259656 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb418100bd0 0x7fb418195960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:44:43.402 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:43.400+0000 7fb41fc93700 1 -- 172.21.15.155:0/938259656 >> 172.21.15.155:0/938259656 conn(0x7fb4180faf70 msgr2=0x7fb418104220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:44:43.403 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:43.400+0000 7fb41fc93700 1 -- 172.21.15.155:0/938259656 shutdown_connections 2024-03-24T09:44:43.403 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:43.400+0000 7fb41fc93700 1 -- 172.21.15.155:0/938259656 wait complete. 2024-03-24T09:44:43.812 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:43 smithi022 ceph-mon[29512]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T09:44:43.812 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:43 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:43.812 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:43 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:43.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:43 smithi155 ceph-mon[37016]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T09:44:43.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:43 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:43.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:43 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:44.280 DEBUG:teuthology.orchestra.run.smithi155:osd.4> sudo journalctl -f -n 0 -u ceph-45356448-e9c2-11ee-95cd-87774f69a715@osd.4.service 2024-03-24T09:44:44.283 INFO:tasks.cephadm:Deploying osd.5 on smithi155 with /dev/vg_nvme/lv_2... 2024-03-24T09:44:44.284 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-24T09:44:44.537 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi155/config 2024-03-24T09:44:44.660 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:44 smithi155 ceph-mon[37016]: from='osd.4 [v2:172.21.15.155:6808/2965421758,v1:172.21.15.155:6809/2965421758]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T09:44:44.660 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:44 smithi155 ceph-mon[37016]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T09:44:44.660 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:44 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:44.660 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:44 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:45.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:44 smithi022 ceph-mon[29512]: from='osd.4 [v2:172.21.15.155:6808/2965421758,v1:172.21.15.155:6809/2965421758]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T09:44:45.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:44 smithi022 ceph-mon[29512]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T09:44:45.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:44 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:45.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:44 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:45.708 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:45 smithi155 ceph-mon[37016]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T09:44:45.708 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:45 smithi155 ceph-mon[37016]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T09:44:45.708 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:45 smithi155 ceph-mon[37016]: osdmap e32: 5 total, 4 up, 5 in 2024-03-24T09:44:45.708 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:45 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T09:44:45.708 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:45 smithi155 ceph-mon[37016]: from='osd.4 [v2:172.21.15.155:6808/2965421758,v1:172.21.15.155:6809/2965421758]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-24T09:44:45.708 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:45 smithi155 ceph-mon[37016]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-24T09:44:45.709 INFO:journalctl@ceph.osd.4.smithi155.stdout:Mar 24 09:44:45 smithi155 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-4[50015]: 2024-03-24T09:44:45.548+0000 7fe2b3a1f700 -1 osd.4 0 waiting for initial osdmap 2024-03-24T09:44:45.709 INFO:journalctl@ceph.osd.4.smithi155.stdout:Mar 24 09:44:45 smithi155 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-4[50015]: 2024-03-24T09:44:45.555+0000 7fe2b0015700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T09:44:45.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:45 smithi022 ceph-mon[29512]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T09:44:45.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:45 smithi022 ceph-mon[29512]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T09:44:45.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:45 smithi022 ceph-mon[29512]: osdmap e32: 5 total, 4 up, 5 in 2024-03-24T09:44:45.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:45 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T09:44:45.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:45 smithi022 ceph-mon[29512]: from='osd.4 [v2:172.21.15.155:6808/2965421758,v1:172.21.15.155:6809/2965421758]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-24T09:44:45.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:45 smithi022 ceph-mon[29512]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-24T09:44:46.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:46 smithi155 ceph-mon[37016]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-03-24T09:44:46.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:46 smithi155 ceph-mon[37016]: osdmap e33: 5 total, 4 up, 5 in 2024-03-24T09:44:46.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:46 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T09:44:46.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:46 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T09:44:47.019 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-24T09:44:47.019 INFO:teuthology.orchestra.run.smithi155.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-24T09:44:47.019 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10+0 records in 2024-03-24T09:44:47.019 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records out 2024-03-24T09:44:47.020 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0133682 s, 784 MB/s 2024-03-24T09:44:47.020 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping successful for: 2024-03-24T09:44:47.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:46 smithi022 ceph-mon[29512]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-03-24T09:44:47.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:46 smithi022 ceph-mon[29512]: osdmap e33: 5 total, 4 up, 5 in 2024-03-24T09:44:47.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:46 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T09:44:47.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:46 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T09:44:47.645 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:47 smithi155 ceph-mon[37016]: purged_snaps scrub starts 2024-03-24T09:44:47.645 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:47 smithi155 ceph-mon[37016]: purged_snaps scrub ok 2024-03-24T09:44:47.645 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:47 smithi155 ceph-mon[37016]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T09:44:47.645 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:47 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T09:44:47.645 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:47 smithi155 ceph-mon[37016]: osd.4 [v2:172.21.15.155:6808/2965421758,v1:172.21.15.155:6809/2965421758] boot 2024-03-24T09:44:47.645 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:47 smithi155 ceph-mon[37016]: osdmap e34: 5 total, 5 up, 5 in 2024-03-24T09:44:47.645 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:47 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T09:44:47.666 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi155:vg_nvme/lv_2 2024-03-24T09:44:48.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:47 smithi022 ceph-mon[29512]: purged_snaps scrub starts 2024-03-24T09:44:48.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:47 smithi022 ceph-mon[29512]: purged_snaps scrub ok 2024-03-24T09:44:48.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:47 smithi022 ceph-mon[29512]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-24T09:44:48.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:47 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T09:44:48.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:47 smithi022 ceph-mon[29512]: osd.4 [v2:172.21.15.155:6808/2965421758,v1:172.21.15.155:6809/2965421758] boot 2024-03-24T09:44:48.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:47 smithi022 ceph-mon[29512]: osdmap e34: 5 total, 5 up, 5 in 2024-03-24T09:44:48.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:47 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T09:44:48.141 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi155/config 2024-03-24T09:44:48.824 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:48 smithi155 ceph-mon[37016]: osdmap e35: 5 total, 5 up, 5 in 2024-03-24T09:44:48.851 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:48 smithi022 ceph-mon[29512]: osdmap e35: 5 total, 5 up, 5 in 2024-03-24T09:44:49.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:49 smithi155 ceph-mon[37016]: pgmap v118: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:44:49.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:49 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:49.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:49 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:49.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:49 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:44:49.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:49 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:44:49.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:49 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:49.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:49 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:44:49.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:49 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:44:49.713 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:49 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:49.854 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:49 smithi022 ceph-mon[29512]: pgmap v118: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:44:49.854 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:49 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:49.854 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:49 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:49.855 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:49 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:44:49.855 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:49 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:44:49.855 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:49 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:49.855 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:49 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:44:49.855 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:49 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:44:49.855 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:49 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:50.481 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:50.477+0000 7fa32fe6c700 1 -- 172.21.15.155:0/2748803120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa328103ea0 msgr2=0x7fa328106290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:44:50.481 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:50.477+0000 7fa32fe6c700 1 --2- 172.21.15.155:0/2748803120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa328103ea0 0x7fa328106290 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa318009910 tx=0x7fa318009c20 comp rx=0 tx=0).stop 2024-03-24T09:44:50.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:50.477+0000 7fa32fe6c700 1 -- 172.21.15.155:0/2748803120 shutdown_connections 2024-03-24T09:44:50.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:50.477+0000 7fa32fe6c700 1 --2- 172.21.15.155:0/2748803120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa328103ea0 0x7fa328106290 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:44:50.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:50.477+0000 7fa32fe6c700 1 --2- 172.21.15.155:0/2748803120 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa328101570 0x7fa328103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:44:50.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:50.477+0000 7fa32fe6c700 1 -- 172.21.15.155:0/2748803120 >> 172.21.15.155:0/2748803120 conn(0x7fa3280faf80 msgr2=0x7fa3280fd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:44:50.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:50.477+0000 7fa32fe6c700 1 -- 172.21.15.155:0/2748803120 shutdown_connections 2024-03-24T09:44:50.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:50.477+0000 7fa32fe6c700 1 -- 172.21.15.155:0/2748803120 wait complete. 2024-03-24T09:44:50.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:50.477+0000 7fa32fe6c700 1 Processor -- start 2024-03-24T09:44:50.483 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:50.477+0000 7fa32fe6c700 1 -- start start 2024-03-24T09:44:50.483 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:50.478+0000 7fa32fe6c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa328101570 0x7fa328100e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:44:50.483 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:50.478+0000 7fa32fe6c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa328103ea0 0x7fa3280ff520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:44:50.483 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:50.478+0000 7fa32fe6c700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3280ffa60 con 0x7fa328103ea0 2024-03-24T09:44:50.483 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:50.478+0000 7fa32fe6c700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3280ffba0 con 0x7fa328101570 2024-03-24T09:44:50.483 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:50.478+0000 7fa32dc08700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa328101570 0x7fa328100e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:44:50.483 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:50.478+0000 7fa32dc08700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa328101570 0x7fa328100e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:56478/0 (socket says 172.21.15.155:56478) 2024-03-24T09:44:50.483 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:50.478+0000 7fa32dc08700 1 -- 172.21.15.155:0/4238295538 learned_addr learned my addr 172.21.15.155:0/4238295538 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-24T09:44:50.484 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:50.478+0000 7fa32dc08700 1 -- 172.21.15.155:0/4238295538 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa328103ea0 msgr2=0x7fa3280ff520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:44:50.484 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:50.478+0000 7fa32d407700 1 --2- 172.21.15.155:0/4238295538 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa328103ea0 0x7fa3280ff520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:44:50.484 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:50.478+0000 7fa32dc08700 1 --2- 172.21.15.155:0/4238295538 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa328103ea0 0x7fa3280ff520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:44:50.484 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:50.478+0000 7fa32dc08700 1 -- 172.21.15.155:0/4238295538 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3180095f0 con 0x7fa328101570 2024-03-24T09:44:50.484 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:50.478+0000 7fa32dc08700 1 --2- 172.21.15.155:0/4238295538 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa328101570 0x7fa328100e80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa32400d8d0 tx=0x7fa32400dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:44:50.484 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:50.478+0000 7fa31effd700 1 -- 172.21.15.155:0/4238295538 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa324009940 con 0x7fa328101570 2024-03-24T09:44:50.484 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:50.478+0000 7fa31effd700 1 -- 172.21.15.155:0/4238295538 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa324010460 con 0x7fa328101570 2024-03-24T09:44:50.484 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:50.478+0000 7fa32fe6c700 1 -- 172.21.15.155:0/4238295538 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3280ffe80 con 0x7fa328101570 2024-03-24T09:44:50.484 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:50.478+0000 7fa31effd700 1 -- 172.21.15.155:0/4238295538 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa32400f5f0 con 0x7fa328101570 2024-03-24T09:44:50.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:50.478+0000 7fa32fe6c700 1 -- 172.21.15.155:0/4238295538 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa32819e510 con 0x7fa328101570 2024-03-24T09:44:50.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:50.479+0000 7fa32fe6c700 1 -- 172.21.15.155:0/4238295538 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa32804ea80 con 0x7fa328101570 2024-03-24T09:44:50.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:50.480+0000 7fa31effd700 1 -- 172.21.15.155:0/4238295538 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7fa3240105d0 con 0x7fa328101570 2024-03-24T09:44:50.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:50.480+0000 7fa31effd700 1 --2- 172.21.15.155:0/4238295538 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fa31406cce0 0x7fa31406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:44:50.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:50.480+0000 7fa31effd700 1 -- 172.21.15.155:0/4238295538 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4206+0+0 (secure 0 0 0) 0x7fa32408bc90 con 0x7fa328101570 2024-03-24T09:44:50.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:50.480+0000 7fa32d407700 1 --2- 172.21.15.155:0/4238295538 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fa31406cce0 0x7fa31406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:44:50.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:50.481+0000 7fa32d407700 1 --2- 172.21.15.155:0/4238295538 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fa31406cce0 0x7fa31406f1a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa31800b5c0 tx=0x7fa318005160 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:44:50.486 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:50.485+0000 7fa31effd700 1 -- 172.21.15.155:0/4238295538 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa32405a060 con 0x7fa328101570 2024-03-24T09:44:50.686 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:44:50.686+0000 7fa32fe6c700 1 -- 172.21.15.155:0/4238295538 --> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fa328105d90 con 0x7fa31406cce0 2024-03-24T09:44:50.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:50 smithi155 ceph-mon[37016]: Detected new or changed devices on smithi155 2024-03-24T09:44:50.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:50 smithi155 ceph-mon[37016]: Adjusting osd_memory_target on smithi155 to 7456M 2024-03-24T09:44:50.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:50 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:44:50.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:50 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:44:50.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:50 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:44:50.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:50 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:51.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:50 smithi022 ceph-mon[29512]: Detected new or changed devices on smithi155 2024-03-24T09:44:51.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:50 smithi022 ceph-mon[29512]: Adjusting osd_memory_target on smithi155 to 7456M 2024-03-24T09:44:51.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:50 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:44:51.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:50 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:44:51.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:50 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:44:51.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:50 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:44:51.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:51 smithi155 ceph-mon[37016]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:44:51.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:51 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:44:51.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:51 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:44:51.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:51 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:44:52.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:51 smithi022 ceph-mon[29512]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:44:52.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:51 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:44:52.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:51 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:44:52.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:51 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:44:52.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:52 smithi155 ceph-mon[37016]: from='client.24173 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:44:53.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:52 smithi022 ceph-mon[29512]: from='client.24173 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:44:53.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:53 smithi155 ceph-mon[37016]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:44:53.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:53 smithi155 ceph-mon[37016]: from='client.? 172.21.15.155:0/1323992009' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa6a8e29-123e-4b1f-8bed-8b207cfadce5"}]: dispatch 2024-03-24T09:44:53.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:53 smithi155 ceph-mon[37016]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa6a8e29-123e-4b1f-8bed-8b207cfadce5"}]: dispatch 2024-03-24T09:44:53.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:53 smithi155 ceph-mon[37016]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa6a8e29-123e-4b1f-8bed-8b207cfadce5"}]': finished 2024-03-24T09:44:53.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:53 smithi155 ceph-mon[37016]: osdmap e36: 6 total, 5 up, 6 in 2024-03-24T09:44:53.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:53 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T09:44:53.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:53 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:44:53.889 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:53 smithi022 ceph-mon[29512]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:44:53.889 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:53 smithi022 ceph-mon[29512]: from='client.? 172.21.15.155:0/1323992009' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa6a8e29-123e-4b1f-8bed-8b207cfadce5"}]: dispatch 2024-03-24T09:44:53.889 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:53 smithi022 ceph-mon[29512]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa6a8e29-123e-4b1f-8bed-8b207cfadce5"}]: dispatch 2024-03-24T09:44:53.890 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:53 smithi022 ceph-mon[29512]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa6a8e29-123e-4b1f-8bed-8b207cfadce5"}]': finished 2024-03-24T09:44:53.890 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:53 smithi022 ceph-mon[29512]: osdmap e36: 6 total, 5 up, 6 in 2024-03-24T09:44:53.890 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:53 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T09:44:53.890 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:53 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:44:54.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:54 smithi155 ceph-mon[37016]: from='client.? 172.21.15.155:0/996808652' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:44:54.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:54 smithi155 ceph-mon[37016]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:44:55.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:54 smithi022 ceph-mon[29512]: from='client.? 172.21.15.155:0/996808652' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:44:55.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:54 smithi022 ceph-mon[29512]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:44:57.649 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:57 smithi155 ceph-mon[37016]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:44:58.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:57 smithi022 ceph-mon[29512]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:44:59.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:44:59 smithi155 ceph-mon[37016]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:44:59.854 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:44:59 smithi022 ceph-mon[29512]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:45:01.211 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:01 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T09:45:01.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:01 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:45:01.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:01 smithi155 ceph-mon[37016]: Deploying daemon osd.5 on smithi155 2024-03-24T09:45:01.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:01 smithi155 ceph-mon[37016]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:45:01.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:01 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T09:45:01.533 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:01 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:45:01.533 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:01 smithi022 ceph-mon[29512]: Deploying daemon osd.5 on smithi155 2024-03-24T09:45:01.533 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:01 smithi022 ceph-mon[29512]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:45:03.686 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:03 smithi155 ceph-mon[37016]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:45:03.967 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:03 smithi022 ceph-mon[29512]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:45:05.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:05 smithi155 ceph-mon[37016]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:45:06.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:05 smithi022 ceph-mon[29512]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:45:06.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:06 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:45:06.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:06 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:45:06.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:06 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:45:07.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:06 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:45:07.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:06 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:45:07.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:06 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:45:07.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:07 smithi155 ceph-mon[37016]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:45:08.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:07 smithi022 ceph-mon[29512]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:45:08.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:08 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:45:09.005 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:08 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:45:09.056 INFO:teuthology.orchestra.run.smithi155.stdout:Created osd(s) 5 on host 'smithi155' 2024-03-24T09:45:09.057 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:09.057+0000 7fa31effd700 1 -- 172.21.15.155:0/4238295538 <== mgr.14215 v2:172.21.15.22:6800/773863266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fa328105d90 con 0x7fa31406cce0 2024-03-24T09:45:09.059 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:09.059+0000 7fa32fe6c700 1 -- 172.21.15.155:0/4238295538 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fa31406cce0 msgr2=0x7fa31406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:09.059 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:09.059+0000 7fa32fe6c700 1 --2- 172.21.15.155:0/4238295538 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fa31406cce0 0x7fa31406f1a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa31800b5c0 tx=0x7fa318005160 comp rx=0 tx=0).stop 2024-03-24T09:45:09.059 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:09.059+0000 7fa32fe6c700 1 -- 172.21.15.155:0/4238295538 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa328101570 msgr2=0x7fa328100e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:09.059 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:09.059+0000 7fa32fe6c700 1 --2- 172.21.15.155:0/4238295538 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa328101570 0x7fa328100e80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa32400d8d0 tx=0x7fa32400dc90 comp rx=0 tx=0).stop 2024-03-24T09:45:09.059 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:09.059+0000 7fa32fe6c700 1 -- 172.21.15.155:0/4238295538 shutdown_connections 2024-03-24T09:45:09.059 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:09.059+0000 7fa32fe6c700 1 --2- 172.21.15.155:0/4238295538 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fa31406cce0 0x7fa31406f1a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:09.059 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:09.059+0000 7fa32fe6c700 1 --2- 172.21.15.155:0/4238295538 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa328101570 0x7fa328100e80 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:09.059 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:09.059+0000 7fa32fe6c700 1 --2- 172.21.15.155:0/4238295538 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa328103ea0 0x7fa3280ff520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:09.059 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:09.059+0000 7fa32fe6c700 1 -- 172.21.15.155:0/4238295538 >> 172.21.15.155:0/4238295538 conn(0x7fa3280faf80 msgr2=0x7fa3280fd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:45:09.059 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:09.059+0000 7fa32fe6c700 1 -- 172.21.15.155:0/4238295538 shutdown_connections 2024-03-24T09:45:09.060 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:09.059+0000 7fa32fe6c700 1 -- 172.21.15.155:0/4238295538 wait complete. 2024-03-24T09:45:09.670 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:09 smithi155 ceph-mon[37016]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:45:09.670 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:09 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:45:09.670 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:09 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:45:09.690 DEBUG:teuthology.orchestra.run.smithi155:osd.5> sudo journalctl -f -n 0 -u ceph-45356448-e9c2-11ee-95cd-87774f69a715@osd.5.service 2024-03-24T09:45:09.693 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-03-24T09:45:09.693 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-24T09:45:09.855 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:09 smithi022 ceph-mon[29512]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:45:09.855 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:09 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:45:09.855 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:09 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:45:09.915 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:45:10.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:10 smithi022 ceph-mon[29512]: from='osd.5 [v2:172.21.15.155:6816/3664800012,v1:172.21.15.155:6817/3664800012]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T09:45:10.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:10 smithi022 ceph-mon[29512]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T09:45:10.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:10 smithi155 ceph-mon[37016]: from='osd.5 [v2:172.21.15.155:6816/3664800012,v1:172.21.15.155:6817/3664800012]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T09:45:10.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:10 smithi155 ceph-mon[37016]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T09:45:11.642 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:11 smithi022 ceph-mon[29512]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:45:11.642 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:11 smithi022 ceph-mon[29512]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T09:45:11.643 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:11 smithi022 ceph-mon[29512]: osdmap e37: 6 total, 5 up, 6 in 2024-03-24T09:45:11.643 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:11 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T09:45:11.643 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:11 smithi022 ceph-mon[29512]: from='osd.5 [v2:172.21.15.155:6816/3664800012,v1:172.21.15.155:6817/3664800012]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-24T09:45:11.643 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:11 smithi022 ceph-mon[29512]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-24T09:45:11.643 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:11 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:45:11.643 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:11 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:45:11.745 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:11 smithi155 ceph-mon[37016]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:45:11.746 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:11 smithi155 ceph-mon[37016]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T09:45:11.746 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:11 smithi155 ceph-mon[37016]: osdmap e37: 6 total, 5 up, 6 in 2024-03-24T09:45:11.746 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:11 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T09:45:11.746 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:11 smithi155 ceph-mon[37016]: from='osd.5 [v2:172.21.15.155:6816/3664800012,v1:172.21.15.155:6817/3664800012]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-24T09:45:11.746 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:11 smithi155 ceph-mon[37016]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-24T09:45:11.746 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:11 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:45:11.746 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:11 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:45:11.747 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 24 09:45:11 smithi155 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-5[56994]: 2024-03-24T09:45:11.574+0000 7fcc42693700 -1 osd.5 0 waiting for initial osdmap 2024-03-24T09:45:11.747 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 24 09:45:11 smithi155 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-5[56994]: 2024-03-24T09:45:11.586+0000 7fcc3dc87700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T09:45:11.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.744+0000 7fdbbf425700 1 -- 172.21.15.22:0/3377572617 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdbb8100b10 msgr2=0x7fdbb8100f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:11.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.744+0000 7fdbbf425700 1 --2- 172.21.15.22:0/3377572617 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdbb8100b10 0x7fdbb8100f30 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fdbac009910 tx=0x7fdbac009c20 comp rx=0 tx=0).stop 2024-03-24T09:45:11.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.744+0000 7fdbbf425700 1 -- 172.21.15.22:0/3377572617 shutdown_connections 2024-03-24T09:45:11.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.744+0000 7fdbbf425700 1 --2- 172.21.15.22:0/3377572617 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbb8101d10 0x7fdbb8102170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:11.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.744+0000 7fdbbf425700 1 --2- 172.21.15.22:0/3377572617 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdbb8100b10 0x7fdbb8100f30 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:11.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.744+0000 7fdbbf425700 1 -- 172.21.15.22:0/3377572617 >> 172.21.15.22:0/3377572617 conn(0x7fdbb80fc0d0 msgr2=0x7fdbb80fe4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:45:11.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.744+0000 7fdbbf425700 1 -- 172.21.15.22:0/3377572617 shutdown_connections 2024-03-24T09:45:11.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.744+0000 7fdbbf425700 1 -- 172.21.15.22:0/3377572617 wait complete. 2024-03-24T09:45:11.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.745+0000 7fdbbf425700 1 Processor -- start 2024-03-24T09:45:11.750 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.745+0000 7fdbbf425700 1 -- start start 2024-03-24T09:45:11.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.745+0000 7fdbbf425700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbb8100b10 0x7fdbb81964f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:11.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.745+0000 7fdbbf425700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdbb8101d10 0x7fdbb8196a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:11.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.745+0000 7fdbbf425700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb8197050 con 0x7fdbb8101d10 2024-03-24T09:45:11.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.745+0000 7fdbbf425700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb8197190 con 0x7fdbb8100b10 2024-03-24T09:45:11.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.745+0000 7fdbbc9c0700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdbb8101d10 0x7fdbb8196a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:11.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.745+0000 7fdbbc9c0700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdbb8101d10 0x7fdbb8196a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:39278/0 (socket says 172.21.15.22:39278) 2024-03-24T09:45:11.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.745+0000 7fdbbc9c0700 1 -- 172.21.15.22:0/2611703708 learned_addr learned my addr 172.21.15.22:0/2611703708 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:45:11.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.745+0000 7fdbbd1c1700 1 --2- 172.21.15.22:0/2611703708 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbb8100b10 0x7fdbb81964f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:11.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.745+0000 7fdbbc9c0700 1 -- 172.21.15.22:0/2611703708 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbb8100b10 msgr2=0x7fdbb81964f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:11.751 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.745+0000 7fdbbc9c0700 1 --2- 172.21.15.22:0/2611703708 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbb8100b10 0x7fdbb81964f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:11.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.745+0000 7fdbbc9c0700 1 -- 172.21.15.22:0/2611703708 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbac0095f0 con 0x7fdbb8101d10 2024-03-24T09:45:11.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.746+0000 7fdbbc9c0700 1 --2- 172.21.15.22:0/2611703708 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdbb8101d10 0x7fdbb8196a30 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fdbb400d8d0 tx=0x7fdbb400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:11.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.746+0000 7fdbaa7fc700 1 -- 172.21.15.22:0/2611703708 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdbb4009940 con 0x7fdbb8101d10 2024-03-24T09:45:11.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.746+0000 7fdbaa7fc700 1 -- 172.21.15.22:0/2611703708 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdbb4010460 con 0x7fdbb8101d10 2024-03-24T09:45:11.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.746+0000 7fdbbf425700 1 -- 172.21.15.22:0/2611703708 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbb819bbf0 con 0x7fdbb8101d10 2024-03-24T09:45:11.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.746+0000 7fdbaa7fc700 1 -- 172.21.15.22:0/2611703708 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdbb400f5f0 con 0x7fdbb8101d10 2024-03-24T09:45:11.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.746+0000 7fdbbf425700 1 -- 172.21.15.22:0/2611703708 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbb819c1c0 con 0x7fdbb8101d10 2024-03-24T09:45:11.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.747+0000 7fdbbf425700 1 -- 172.21.15.22:0/2611703708 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbb8190690 con 0x7fdbb8101d10 2024-03-24T09:45:11.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.747+0000 7fdbaa7fc700 1 -- 172.21.15.22:0/2611703708 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7fdbb40105d0 con 0x7fdbb8101d10 2024-03-24T09:45:11.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.748+0000 7fdbaa7fc700 1 --2- 172.21.15.22:0/2611703708 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fdba406cce0 0x7fdba406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:11.752 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.748+0000 7fdbaa7fc700 1 -- 172.21.15.22:0/2611703708 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4358+0+0 (secure 0 0 0) 0x7fdbb408be30 con 0x7fdbb8101d10 2024-03-24T09:45:11.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.748+0000 7fdbbd1c1700 1 --2- 172.21.15.22:0/2611703708 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fdba406cce0 0x7fdba406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:11.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.748+0000 7fdbbd1c1700 1 --2- 172.21.15.22:0/2611703708 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fdba406cce0 0x7fdba406f1a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdbac006010 tx=0x7fdbac00b580 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:11.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.753+0000 7fdbaa7fc700 1 -- 172.21.15.22:0/2611703708 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fdbb405a0e0 con 0x7fdbb8101d10 2024-03-24T09:45:11.954 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.953+0000 7fdbbf425700 1 -- 172.21.15.22:0/2611703708 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fdbb80619c0 con 0x7fdbb8101d10 2024-03-24T09:45:11.954 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.953+0000 7fdbaa7fc700 1 -- 172.21.15.22:0/2611703708 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v38) v1 ==== 74+0+130 (secure 0 0 0) 0x7fdbb4059c70 con 0x7fdbb8101d10 2024-03-24T09:45:11.954 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:45:11.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.955+0000 7fdbbf425700 1 -- 172.21.15.22:0/2611703708 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fdba406cce0 msgr2=0x7fdba406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:11.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.955+0000 7fdbbf425700 1 --2- 172.21.15.22:0/2611703708 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fdba406cce0 0x7fdba406f1a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdbac006010 tx=0x7fdbac00b580 comp rx=0 tx=0).stop 2024-03-24T09:45:11.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.955+0000 7fdbbf425700 1 -- 172.21.15.22:0/2611703708 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdbb8101d10 msgr2=0x7fdbb8196a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:11.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.955+0000 7fdbbf425700 1 --2- 172.21.15.22:0/2611703708 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdbb8101d10 0x7fdbb8196a30 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fdbb400d8d0 tx=0x7fdbb400dc90 comp rx=0 tx=0).stop 2024-03-24T09:45:11.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.955+0000 7fdbbf425700 1 -- 172.21.15.22:0/2611703708 shutdown_connections 2024-03-24T09:45:11.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.955+0000 7fdbbf425700 1 --2- 172.21.15.22:0/2611703708 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fdba406cce0 0x7fdba406f1a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:11.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.955+0000 7fdbbf425700 1 --2- 172.21.15.22:0/2611703708 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbb8100b10 0x7fdbb81964f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:11.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.955+0000 7fdbbf425700 1 --2- 172.21.15.22:0/2611703708 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdbb8101d10 0x7fdbb8196a30 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:11.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.955+0000 7fdbbf425700 1 -- 172.21.15.22:0/2611703708 >> 172.21.15.22:0/2611703708 conn(0x7fdbb80fc0d0 msgr2=0x7fdbb8105310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:45:11.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.956+0000 7fdbbf425700 1 -- 172.21.15.22:0/2611703708 shutdown_connections 2024-03-24T09:45:11.957 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:11.956+0000 7fdbbf425700 1 -- 172.21.15.22:0/2611703708 wait complete. 2024-03-24T09:45:12.618 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":38,"num_osds":6,"num_up_osds":5,"osd_up_since":1711273486,"num_in_osds":6,"osd_in_since":1711273493,"num_remapped_pgs":0} 2024-03-24T09:45:12.863 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:12 smithi155 ceph-mon[37016]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-03-24T09:45:12.863 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:12 smithi155 ceph-mon[37016]: osdmap e38: 6 total, 5 up, 6 in 2024-03-24T09:45:12.863 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:12 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T09:45:12.863 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:12 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T09:45:12.863 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:12 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/2611703708' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T09:45:12.882 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:12 smithi022 ceph-mon[29512]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-03-24T09:45:12.883 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:12 smithi022 ceph-mon[29512]: osdmap e38: 6 total, 5 up, 6 in 2024-03-24T09:45:12.883 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:12 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T09:45:12.883 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:12 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T09:45:12.883 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:12 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/2611703708' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T09:45:13.620 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-24T09:45:13.839 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:45:13.863 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:13 smithi022 ceph-mon[29512]: purged_snaps scrub starts 2024-03-24T09:45:13.864 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:13 smithi022 ceph-mon[29512]: purged_snaps scrub ok 2024-03-24T09:45:13.864 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:13 smithi022 ceph-mon[29512]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:45:13.864 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:13 smithi022 ceph-mon[29512]: osd.5 [v2:172.21.15.155:6816/3664800012,v1:172.21.15.155:6817/3664800012] boot 2024-03-24T09:45:13.864 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:13 smithi022 ceph-mon[29512]: osdmap e39: 6 total, 6 up, 6 in 2024-03-24T09:45:13.864 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:13 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T09:45:13.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:13 smithi155 ceph-mon[37016]: purged_snaps scrub starts 2024-03-24T09:45:13.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:13 smithi155 ceph-mon[37016]: purged_snaps scrub ok 2024-03-24T09:45:13.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:13 smithi155 ceph-mon[37016]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:45:13.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:13 smithi155 ceph-mon[37016]: osd.5 [v2:172.21.15.155:6816/3664800012,v1:172.21.15.155:6817/3664800012] boot 2024-03-24T09:45:13.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:13 smithi155 ceph-mon[37016]: osdmap e39: 6 total, 6 up, 6 in 2024-03-24T09:45:13.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:13 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T09:45:14.917 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:14 smithi155 ceph-mon[37016]: osdmap e40: 6 total, 6 up, 6 in 2024-03-24T09:45:14.917 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:14 smithi155 ceph-mon[37016]: Detected new or changed devices on smithi155 2024-03-24T09:45:14.917 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:14 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:45:14.917 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:14 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:45:14.917 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:14 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:45:14.917 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:14 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:45:14.917 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:14 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:45:14.917 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:14 smithi155 ceph-mon[37016]: Adjusting osd_memory_target on smithi155 to 4970M 2024-03-24T09:45:14.918 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:14 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:45:14.918 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:14 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:45:14.918 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:14 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:45:14.918 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:14 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:45:14.918 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:14 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:45:14.918 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:14 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:45:14.918 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:14 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:45:14.918 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:14 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:45:14.918 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:14 smithi155 ceph-mon[37016]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:14.974 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:14 smithi022 ceph-mon[29512]: osdmap e40: 6 total, 6 up, 6 in 2024-03-24T09:45:14.974 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:14 smithi022 ceph-mon[29512]: Detected new or changed devices on smithi155 2024-03-24T09:45:14.974 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:14 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:45:14.975 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:14 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:45:14.975 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:14 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:45:14.975 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:14 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:45:14.975 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:14 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:45:14.975 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:14 smithi022 ceph-mon[29512]: Adjusting osd_memory_target on smithi155 to 4970M 2024-03-24T09:45:14.975 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:14 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:45:14.975 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:14 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:45:14.975 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:14 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:45:14.976 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:14 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:45:14.976 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:14 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:45:14.976 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:14 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:45:14.976 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:14 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:45:14.976 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:14 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:45:14.976 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:14 smithi022 ceph-mon[29512]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:15.535 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.532+0000 7f3b67871700 1 -- 172.21.15.22:0/3936379833 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3b600737b0 msgr2=0x7f3b60073c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:15.535 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.532+0000 7f3b67871700 1 --2- 172.21.15.22:0/3936379833 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3b600737b0 0x7f3b60073c30 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3b50009910 tx=0x7f3b50009c20 comp rx=0 tx=0).stop 2024-03-24T09:45:15.535 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.533+0000 7f3b67871700 1 -- 172.21.15.22:0/3936379833 shutdown_connections 2024-03-24T09:45:15.535 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.533+0000 7f3b67871700 1 --2- 172.21.15.22:0/3936379833 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3b600737b0 0x7f3b60073c30 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:15.535 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.533+0000 7f3b67871700 1 --2- 172.21.15.22:0/3936379833 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3b60074d80 0x7f3b600731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:15.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.533+0000 7f3b67871700 1 -- 172.21.15.22:0/3936379833 >> 172.21.15.22:0/3936379833 conn(0x7f3b600fc0a0 msgr2=0x7f3b600fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:45:15.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.533+0000 7f3b67871700 1 -- 172.21.15.22:0/3936379833 shutdown_connections 2024-03-24T09:45:15.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.533+0000 7f3b67871700 1 -- 172.21.15.22:0/3936379833 wait complete. 2024-03-24T09:45:15.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.533+0000 7f3b67871700 1 Processor -- start 2024-03-24T09:45:15.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.533+0000 7f3b67871700 1 -- start start 2024-03-24T09:45:15.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.534+0000 7f3b67871700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3b600737b0 0x7f3b601987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:15.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.534+0000 7f3b67871700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3b60074d80 0x7f3b60198d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:15.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.534+0000 7f3b67871700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b60199320 con 0x7f3b60074d80 2024-03-24T09:45:15.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.534+0000 7f3b67871700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b60199460 con 0x7f3b600737b0 2024-03-24T09:45:15.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.534+0000 7f3b64e0c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3b60074d80 0x7f3b60198d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:15.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.534+0000 7f3b64e0c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3b60074d80 0x7f3b60198d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:39298/0 (socket says 172.21.15.22:39298) 2024-03-24T09:45:15.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.534+0000 7f3b64e0c700 1 -- 172.21.15.22:0/3568937988 learned_addr learned my addr 172.21.15.22:0/3568937988 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:45:15.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.534+0000 7f3b6560d700 1 --2- 172.21.15.22:0/3568937988 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3b600737b0 0x7f3b601987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:15.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.534+0000 7f3b64e0c700 1 -- 172.21.15.22:0/3568937988 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3b600737b0 msgr2=0x7f3b601987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:15.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.534+0000 7f3b64e0c700 1 --2- 172.21.15.22:0/3568937988 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3b600737b0 0x7f3b601987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:15.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.534+0000 7f3b64e0c700 1 -- 172.21.15.22:0/3568937988 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b500095f0 con 0x7f3b60074d80 2024-03-24T09:45:15.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.534+0000 7f3b64e0c700 1 --2- 172.21.15.22:0/3568937988 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3b60074d80 0x7f3b60198d00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3b500092e0 tx=0x7f3b5000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:15.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.534+0000 7f3b567fc700 1 -- 172.21.15.22:0/3568937988 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b5001d070 con 0x7f3b60074d80 2024-03-24T09:45:15.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.534+0000 7f3b67871700 1 -- 172.21.15.22:0/3568937988 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b6019de10 con 0x7f3b60074d80 2024-03-24T09:45:15.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.534+0000 7f3b67871700 1 -- 172.21.15.22:0/3568937988 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b6019e350 con 0x7f3b60074d80 2024-03-24T09:45:15.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.534+0000 7f3b567fc700 1 -- 172.21.15.22:0/3568937988 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3b5000f460 con 0x7f3b60074d80 2024-03-24T09:45:15.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.534+0000 7f3b567fc700 1 -- 172.21.15.22:0/3568937988 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b50021640 con 0x7f3b60074d80 2024-03-24T09:45:15.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.535+0000 7f3b67871700 1 -- 172.21.15.22:0/3568937988 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b601928d0 con 0x7f3b60074d80 2024-03-24T09:45:15.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.536+0000 7f3b567fc700 1 -- 172.21.15.22:0/3568937988 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f3b5000f5d0 con 0x7f3b60074d80 2024-03-24T09:45:15.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.536+0000 7f3b567fc700 1 --2- 172.21.15.22:0/3568937988 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f3b4c06cce0 0x7f3b4c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:15.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.536+0000 7f3b567fc700 1 -- 172.21.15.22:0/3568937988 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f3b5008d1b0 con 0x7f3b60074d80 2024-03-24T09:45:15.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.536+0000 7f3b6560d700 1 --2- 172.21.15.22:0/3568937988 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f3b4c06cce0 0x7f3b4c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:15.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.537+0000 7f3b6560d700 1 --2- 172.21.15.22:0/3568937988 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f3b4c06cce0 0x7f3b4c06f1a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c006fd0 tx=0x7f3b5c008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:15.541 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.541+0000 7f3b567fc700 1 -- 172.21.15.22:0/3568937988 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3b5005b3d0 con 0x7f3b60074d80 2024-03-24T09:45:15.744 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.744+0000 7f3b67871700 1 -- 172.21.15.22:0/3568937988 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f3b6004ea80 con 0x7f3b60074d80 2024-03-24T09:45:15.744 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.744+0000 7f3b567fc700 1 -- 172.21.15.22:0/3568937988 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v40) v1 ==== 74+0+130 (secure 0 0 0) 0x7f3b5005af60 con 0x7f3b60074d80 2024-03-24T09:45:15.744 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:45:15.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.746+0000 7f3b67871700 1 -- 172.21.15.22:0/3568937988 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f3b4c06cce0 msgr2=0x7f3b4c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:15.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.746+0000 7f3b67871700 1 --2- 172.21.15.22:0/3568937988 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f3b4c06cce0 0x7f3b4c06f1a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c006fd0 tx=0x7f3b5c008040 comp rx=0 tx=0).stop 2024-03-24T09:45:15.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.746+0000 7f3b67871700 1 -- 172.21.15.22:0/3568937988 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3b60074d80 msgr2=0x7f3b60198d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:15.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.746+0000 7f3b67871700 1 --2- 172.21.15.22:0/3568937988 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3b60074d80 0x7f3b60198d00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3b500092e0 tx=0x7f3b5000be20 comp rx=0 tx=0).stop 2024-03-24T09:45:15.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.746+0000 7f3b67871700 1 -- 172.21.15.22:0/3568937988 shutdown_connections 2024-03-24T09:45:15.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.746+0000 7f3b67871700 1 --2- 172.21.15.22:0/3568937988 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f3b4c06cce0 0x7f3b4c06f1a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:15.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.746+0000 7f3b67871700 1 --2- 172.21.15.22:0/3568937988 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3b600737b0 0x7f3b601987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:15.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.746+0000 7f3b67871700 1 --2- 172.21.15.22:0/3568937988 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3b60074d80 0x7f3b60198d00 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:15.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.746+0000 7f3b67871700 1 -- 172.21.15.22:0/3568937988 >> 172.21.15.22:0/3568937988 conn(0x7f3b600fc0a0 msgr2=0x7f3b60106d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:45:15.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.746+0000 7f3b67871700 1 -- 172.21.15.22:0/3568937988 shutdown_connections 2024-03-24T09:45:15.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:15.746+0000 7f3b67871700 1 -- 172.21.15.22:0/3568937988 wait complete. 2024-03-24T09:45:16.426 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":40,"num_osds":6,"num_up_osds":6,"osd_up_since":1711273512,"num_in_osds":6,"osd_in_since":1711273493,"num_remapped_pgs":0} 2024-03-24T09:45:16.427 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T09:45:16.639 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:45:16.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:16 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/3568937988' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T09:45:16.760 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:16 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/3568937988' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T09:45:17.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:17 smithi155 ceph-mon[37016]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:17.781 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:17 smithi022 ceph-mon[29512]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:18.306 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.304+0000 7f0a87b2a700 1 -- 172.21.15.22:0/4143268473 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0a801059a0 msgr2=0x7f0a80105d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:18.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.304+0000 7f0a87b2a700 1 --2- 172.21.15.22:0/4143268473 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0a801059a0 0x7f0a80105d80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c009910 tx=0x7f0a7c009c20 comp rx=0 tx=0).stop 2024-03-24T09:45:18.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.305+0000 7f0a87b2a700 1 -- 172.21.15.22:0/4143268473 shutdown_connections 2024-03-24T09:45:18.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.305+0000 7f0a87b2a700 1 --2- 172.21.15.22:0/4143268473 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a800ff9a0 0x7f0a800ffe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:18.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.305+0000 7f0a87b2a700 1 --2- 172.21.15.22:0/4143268473 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0a801059a0 0x7f0a80105d80 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:18.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.305+0000 7f0a87b2a700 1 -- 172.21.15.22:0/4143268473 >> 172.21.15.22:0/4143268473 conn(0x7f0a800fb4b0 msgr2=0x7f0a800fd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:45:18.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.305+0000 7f0a87b2a700 1 -- 172.21.15.22:0/4143268473 shutdown_connections 2024-03-24T09:45:18.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.305+0000 7f0a87b2a700 1 -- 172.21.15.22:0/4143268473 wait complete. 2024-03-24T09:45:18.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.305+0000 7f0a87b2a700 1 Processor -- start 2024-03-24T09:45:18.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.305+0000 7f0a87b2a700 1 -- start start 2024-03-24T09:45:18.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.305+0000 7f0a87b2a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a800ff9a0 0x7f0a80197820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:18.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.305+0000 7f0a87b2a700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0a801059a0 0x7f0a80197d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:18.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.305+0000 7f0a87b2a700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a801982a0 con 0x7f0a801059a0 2024-03-24T09:45:18.307 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.305+0000 7f0a87b2a700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a801983e0 con 0x7f0a800ff9a0 2024-03-24T09:45:18.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.305+0000 7f0a850c5700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0a801059a0 0x7f0a80197d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:18.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.306+0000 7f0a850c5700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0a801059a0 0x7f0a80197d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:39322/0 (socket says 172.21.15.22:39322) 2024-03-24T09:45:18.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.306+0000 7f0a850c5700 1 -- 172.21.15.22:0/3476443443 learned_addr learned my addr 172.21.15.22:0/3476443443 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:45:18.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.306+0000 7f0a858c6700 1 --2- 172.21.15.22:0/3476443443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a800ff9a0 0x7f0a80197820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:18.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.306+0000 7f0a850c5700 1 -- 172.21.15.22:0/3476443443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a800ff9a0 msgr2=0x7f0a80197820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:18.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.306+0000 7f0a850c5700 1 --2- 172.21.15.22:0/3476443443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a800ff9a0 0x7f0a80197820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:18.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.306+0000 7f0a850c5700 1 -- 172.21.15.22:0/3476443443 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a7c0095f0 con 0x7f0a801059a0 2024-03-24T09:45:18.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.306+0000 7f0a858c6700 1 --2- 172.21.15.22:0/3476443443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a800ff9a0 0x7f0a80197820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T09:45:18.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.306+0000 7f0a850c5700 1 --2- 172.21.15.22:0/3476443443 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0a801059a0 0x7f0a80197d60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0a7000c930 tx=0x7f0a7000ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:18.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.306+0000 7f0a76ffd700 1 -- 172.21.15.22:0/3476443443 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a70007ad0 con 0x7f0a801059a0 2024-03-24T09:45:18.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.306+0000 7f0a87b2a700 1 -- 172.21.15.22:0/3476443443 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a801a0890 con 0x7f0a801059a0 2024-03-24T09:45:18.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.306+0000 7f0a76ffd700 1 -- 172.21.15.22:0/3476443443 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0a70007c30 con 0x7f0a801059a0 2024-03-24T09:45:18.308 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.306+0000 7f0a76ffd700 1 -- 172.21.15.22:0/3476443443 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a70018610 con 0x7f0a801059a0 2024-03-24T09:45:18.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.306+0000 7f0a87b2a700 1 -- 172.21.15.22:0/3476443443 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a801a0c40 con 0x7f0a801059a0 2024-03-24T09:45:18.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.307+0000 7f0a87b2a700 1 -- 172.21.15.22:0/3476443443 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a80107fa0 con 0x7f0a801059a0 2024-03-24T09:45:18.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.308+0000 7f0a76ffd700 1 -- 172.21.15.22:0/3476443443 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f0a7000f450 con 0x7f0a801059a0 2024-03-24T09:45:18.309 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.308+0000 7f0a76ffd700 1 --2- 172.21.15.22:0/3476443443 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f0a6c06cce0 0x7f0a6c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:18.310 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.308+0000 7f0a858c6700 1 --2- 172.21.15.22:0/3476443443 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f0a6c06cce0 0x7f0a6c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:18.310 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.308+0000 7f0a76ffd700 1 -- 172.21.15.22:0/3476443443 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f0a7008c0b0 con 0x7f0a801059a0 2024-03-24T09:45:18.310 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.309+0000 7f0a858c6700 1 --2- 172.21.15.22:0/3476443443 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f0a6c06cce0 0x7f0a6c06f1a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c005f50 tx=0x7f0a7c005dc0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:18.313 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.313+0000 7f0a76ffd700 1 -- 172.21.15.22:0/3476443443 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0a700567c0 con 0x7f0a801059a0 2024-03-24T09:45:18.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.515+0000 7f0a87b2a700 1 -- 172.21.15.22:0/3476443443 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0a801010e0 con 0x7f0a801059a0 2024-03-24T09:45:18.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.517+0000 7f0a76ffd700 1 -- 172.21.15.22:0/3476443443 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11348 (secure 0 0 0) 0x7f0a70059de0 con 0x7f0a801059a0 2024-03-24T09:45:18.517 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:45:18.517 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":40,"fsid":"45356448-e9c2-11ee-95cd-87774f69a715","created":"2024-03-24T09:39:25.502234+0000","modified":"2024-03-24T09:45:13.574726+0000","last_up_change":"2024-03-24T09:45:12.567478+0000","last_in_change":"2024-03-24T09:44:53.111067+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T09:43:56.788035+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ba411044-c790-44b5-881e-4e86675a868b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":3431970421},{"type":"v1","addr":"172.21.15.22:6803","nonce":3431970421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":3431970421},{"type":"v1","addr":"172.21.15.22:6805","nonce":3431970421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":3431970421},{"type":"v1","addr":"172.21.15.22:6809","nonce":3431970421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":3431970421},{"type":"v1","addr":"172.21.15.22:6807","nonce":3431970421}]},"public_addr":"172.21.15.22:6803/3431970421","cluster_addr":"172.21.15.22:6805/3431970421","heartbeat_back_addr":"172.21.15.22:6809/3431970421","heartbeat_front_addr":"172.21.15.22:6807/3431970421","state":["exists","up"]},{"osd":1,"uuid":"2bc9ee9b-4d76-4df5-bc42-92f243ba6c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":1675089147},{"type":"v1","addr":"172.21.15.22:6811","nonce":1675089147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":1675089147},{"type":"v1","addr":"172.21.15.22:6813","nonce":1675089147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":1675089147},{"type":"v1","addr":"172.21.15.22:6817","nonce":1675089147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":1675089147},{"type":"v1","addr":"172.21.15.22:6815","nonce":1675089147}]},"public_addr":"172.21.15.22:6811/1675089147","cluster_addr":"172.21.15.22:6813/1675089147","heartbeat_back_addr":"172.21.15.22:6817/1675089147","heartbeat_front_addr":"172.21.15.22:6815/1675089147","state":["exists","up"]},{"osd":2,"uuid":"1e0c3afd-b47b-4de8-ab3c-e25770557c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":1538987315},{"type":"v1","addr":"172.21.15.22:6819","nonce":1538987315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":1538987315},{"type":"v1","addr":"172.21.15.22:6821","nonce":1538987315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":1538987315},{"type":"v1","addr":"172.21.15.22:6825","nonce":1538987315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":1538987315},{"type":"v1","addr":"172.21.15.22:6823","nonce":1538987315}]},"public_addr":"172.21.15.22:6819/1538987315","cluster_addr":"172.21.15.22:6821/1538987315","heartbeat_back_addr":"172.21.15.22:6825/1538987315","heartbeat_front_addr":"172.21.15.22:6823/1538987315","state":["exists","up"]},{"osd":3,"uuid":"56f51449-b4b8-4874-9469-497ce984a5f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":2405415706},{"type":"v1","addr":"172.21.15.155:6801","nonce":2405415706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":2405415706},{"type":"v1","addr":"172.21.15.155:6803","nonce":2405415706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":2405415706},{"type":"v1","addr":"172.21.15.155:6807","nonce":2405415706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":2405415706},{"type":"v1","addr":"172.21.15.155:6805","nonce":2405415706}]},"public_addr":"172.21.15.155:6801/2405415706","cluster_addr":"172.21.15.155:6803/2405415706","heartbeat_back_addr":"172.21.15.155:6807/2405415706","heartbeat_front_addr":"172.21.15.155:6805/2405415706","state":["exists","up"]},{"osd":4,"uuid":"1af86297-1514-495a-8d74-6d0237f82316","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":2965421758},{"type":"v1","addr":"172.21.15.155:6809","nonce":2965421758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":2965421758},{"type":"v1","addr":"172.21.15.155:6811","nonce":2965421758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":2965421758},{"type":"v1","addr":"172.21.15.155:6815","nonce":2965421758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":2965421758},{"type":"v1","addr":"172.21.15.155:6813","nonce":2965421758}]},"public_addr":"172.21.15.155:6809/2965421758","cluster_addr":"172.21.15.155:6811/2965421758","heartbeat_back_addr":"172.21.15.155:6815/2965421758","heartbeat_front_addr":"172.21.15.155:6813/2965421758","state":["exists","up"]},{"osd":5,"uuid":"aa6a8e29-123e-4b1f-8bed-8b207cfadce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3664800012},{"type":"v1","addr":"172.21.15.155:6817","nonce":3664800012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3664800012},{"type":"v1","addr":"172.21.15.155:6819","nonce":3664800012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3664800012},{"type":"v1","addr":"172.21.15.155:6823","nonce":3664800012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3664800012},{"type":"v1","addr":"172.21.15.155:6821","nonce":3664800012}]},"public_addr":"172.21.15.155:6817/3664800012","cluster_addr":"172.21.15.155:6819/3664800012","heartbeat_back_addr":"172.21.15.155:6823/3664800012","heartbeat_front_addr":"172.21.15.155:6821/3664800012","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:42:57.929167+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:43:25.482421+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:43:54.589274+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:44:18.882506+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:44:44.731273+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:45:10.622995+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.22:0/632176108":"2024-03-25T09:41:38.451303+0000","172.21.15.22:6801/2979399337":"2024-03-25T09:41:38.451303+0000","172.21.15.22:0/1096383264":"2024-03-25T09:41:38.451303+0000","172.21.15.22:0/774594813":"2024-03-25T09:40:30.945227+0000","172.21.15.22:6801/3036078499":"2024-03-25T09:40:30.945227+0000","172.21.15.22:6800/3036078499":"2024-03-25T09:40:30.945227+0000","172.21.15.22:0/1182234699":"2024-03-25T09:40:30.945227+0000","172.21.15.22:0/3835020536":"2024-03-25T09:41:38.451303+0000","172.21.15.22:0/2826976179":"2024-03-25T09:39:58.019245+0000","172.21.15.22:0/363626647":"2024-03-25T09:39:58.019245+0000","172.21.15.22:6801/2423771373":"2024-03-25T09:39:58.019245+0000","172.21.15.22:6800/2979399337":"2024-03-25T09:41:38.451303+0000","172.21.15.22:0/2928401592":"2024-03-25T09:39:58.019245+0000","172.21.15.22:6800/2423771373":"2024-03-25T09:39:58.019245+0000","172.21.15.22:0/1186018607":"2024-03-25T09:40:30.945227+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T09:45:18.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.520+0000 7f0a87b2a700 1 -- 172.21.15.22:0/3476443443 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f0a6c06cce0 msgr2=0x7f0a6c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:18.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.520+0000 7f0a87b2a700 1 --2- 172.21.15.22:0/3476443443 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f0a6c06cce0 0x7f0a6c06f1a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c005f50 tx=0x7f0a7c005dc0 comp rx=0 tx=0).stop 2024-03-24T09:45:18.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.520+0000 7f0a87b2a700 1 -- 172.21.15.22:0/3476443443 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0a801059a0 msgr2=0x7f0a80197d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:18.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.520+0000 7f0a87b2a700 1 --2- 172.21.15.22:0/3476443443 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0a801059a0 0x7f0a80197d60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0a7000c930 tx=0x7f0a7000ccf0 comp rx=0 tx=0).stop 2024-03-24T09:45:18.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.520+0000 7f0a87b2a700 1 -- 172.21.15.22:0/3476443443 shutdown_connections 2024-03-24T09:45:18.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.520+0000 7f0a87b2a700 1 --2- 172.21.15.22:0/3476443443 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f0a6c06cce0 0x7f0a6c06f1a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:18.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.520+0000 7f0a87b2a700 1 --2- 172.21.15.22:0/3476443443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a800ff9a0 0x7f0a80197820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:18.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.520+0000 7f0a87b2a700 1 --2- 172.21.15.22:0/3476443443 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0a801059a0 0x7f0a80197d60 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:18.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.520+0000 7f0a87b2a700 1 -- 172.21.15.22:0/3476443443 >> 172.21.15.22:0/3476443443 conn(0x7f0a800fb4b0 msgr2=0x7f0a800fc220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:45:18.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.520+0000 7f0a87b2a700 1 -- 172.21.15.22:0/3476443443 shutdown_connections 2024-03-24T09:45:18.520 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:18.520+0000 7f0a87b2a700 1 -- 172.21.15.22:0/3476443443 wait complete. 2024-03-24T09:45:18.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:18 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/3476443443' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:45:18.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:18 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/3476443443' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:45:19.220 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T09:43:56.788035+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T09:45:19.220 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-24T09:45:19.428 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:45:19.855 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:19 smithi022 ceph-mon[29512]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:19.954 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:19 smithi155 ceph-mon[37016]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:21.060 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.058+0000 7f09aea9b700 1 -- 172.21.15.22:0/1874551719 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f09a8106a70 msgr2=0x7f09a8106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:21.061 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.058+0000 7f09aea9b700 1 --2- 172.21.15.22:0/1874551719 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f09a8106a70 0x7f09a8106e50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f099c009910 tx=0x7f099c009c20 comp rx=0 tx=0).stop 2024-03-24T09:45:21.061 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.058+0000 7f09aea9b700 1 -- 172.21.15.22:0/1874551719 shutdown_connections 2024-03-24T09:45:21.061 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.058+0000 7f09aea9b700 1 --2- 172.21.15.22:0/1874551719 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09a8100a70 0x7f09a8100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:21.061 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.058+0000 7f09aea9b700 1 --2- 172.21.15.22:0/1874551719 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f09a8106a70 0x7f09a8106e50 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:21.061 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.058+0000 7f09aea9b700 1 -- 172.21.15.22:0/1874551719 >> 172.21.15.22:0/1874551719 conn(0x7f09a80fc5a0 msgr2=0x7f09a80fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:45:21.061 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.058+0000 7f09aea9b700 1 -- 172.21.15.22:0/1874551719 shutdown_connections 2024-03-24T09:45:21.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.058+0000 7f09aea9b700 1 -- 172.21.15.22:0/1874551719 wait complete. 2024-03-24T09:45:21.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.058+0000 7f09aea9b700 1 Processor -- start 2024-03-24T09:45:21.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.058+0000 7f09aea9b700 1 -- start start 2024-03-24T09:45:21.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.059+0000 7f09aea9b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09a8100a70 0x7f09a819e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:21.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.059+0000 7f09aea9b700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f09a8106a70 0x7f09a819eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:21.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.059+0000 7f09aea9b700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a819f500 con 0x7f09a8106a70 2024-03-24T09:45:21.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.059+0000 7f09aea9b700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a8198ae0 con 0x7f09a8100a70 2024-03-24T09:45:21.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.059+0000 7f09a7fff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f09a8106a70 0x7f09a819eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:21.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.059+0000 7f09a7fff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f09a8106a70 0x7f09a819eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:45612/0 (socket says 172.21.15.22:45612) 2024-03-24T09:45:21.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.059+0000 7f09a7fff700 1 -- 172.21.15.22:0/584477525 learned_addr learned my addr 172.21.15.22:0/584477525 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:45:21.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.059+0000 7f09ac837700 1 --2- 172.21.15.22:0/584477525 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09a8100a70 0x7f09a819e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:21.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.059+0000 7f09a7fff700 1 -- 172.21.15.22:0/584477525 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09a8100a70 msgr2=0x7f09a819e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:21.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.059+0000 7f09a7fff700 1 --2- 172.21.15.22:0/584477525 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09a8100a70 0x7f09a819e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:21.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.059+0000 7f09a7fff700 1 -- 172.21.15.22:0/584477525 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f099c0095f0 con 0x7f09a8106a70 2024-03-24T09:45:21.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.059+0000 7f09a7fff700 1 --2- 172.21.15.22:0/584477525 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f09a8106a70 0x7f09a819eee0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f099400b4b0 tx=0x7f099400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:21.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.060+0000 7f09a5ffb700 1 -- 172.21.15.22:0/584477525 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0994011800 con 0x7f09a8106a70 2024-03-24T09:45:21.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.060+0000 7f09a5ffb700 1 -- 172.21.15.22:0/584477525 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0994011e20 con 0x7f09a8106a70 2024-03-24T09:45:21.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.060+0000 7f09aea9b700 1 -- 172.21.15.22:0/584477525 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09a8198dc0 con 0x7f09a8106a70 2024-03-24T09:45:21.063 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.060+0000 7f09a5ffb700 1 -- 172.21.15.22:0/584477525 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f099400f2e0 con 0x7f09a8106a70 2024-03-24T09:45:21.064 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.060+0000 7f09aea9b700 1 -- 172.21.15.22:0/584477525 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09a8199310 con 0x7f09a8106a70 2024-03-24T09:45:21.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.061+0000 7f09aea9b700 1 -- 172.21.15.22:0/584477525 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09a8104d20 con 0x7f09a8106a70 2024-03-24T09:45:21.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.061+0000 7f09a5ffb700 1 -- 172.21.15.22:0/584477525 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f0994010380 con 0x7f09a8106a70 2024-03-24T09:45:21.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.062+0000 7f09a5ffb700 1 --2- 172.21.15.22:0/584477525 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f099806cce0 0x7f099806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:21.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.062+0000 7f09a5ffb700 1 -- 172.21.15.22:0/584477525 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f099408bad0 con 0x7f09a8106a70 2024-03-24T09:45:21.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.062+0000 7f09ac837700 1 --2- 172.21.15.22:0/584477525 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f099806cce0 0x7f099806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:21.065 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.062+0000 7f09ac837700 1 --2- 172.21.15.22:0/584477525 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f099806cce0 0x7f099806f1a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f099c005f50 tx=0x7f099c01a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:21.067 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.067+0000 7f09a5ffb700 1 -- 172.21.15.22:0/584477525 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0994059c70 con 0x7f09a8106a70 2024-03-24T09:45:21.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.266+0000 7f09aea9b700 1 -- 172.21.15.22:0/584477525 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f09a8199fa0 con 0x7f09a8106a70 2024-03-24T09:45:21.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.266+0000 7f09a5ffb700 1 -- 172.21.15.22:0/584477525 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0994059800 con 0x7f09a8106a70 2024-03-24T09:45:21.267 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 1 2024-03-24T09:45:21.268 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.268+0000 7f09aea9b700 1 -- 172.21.15.22:0/584477525 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f099806cce0 msgr2=0x7f099806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:21.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.268+0000 7f09aea9b700 1 --2- 172.21.15.22:0/584477525 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f099806cce0 0x7f099806f1a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f099c005f50 tx=0x7f099c01a040 comp rx=0 tx=0).stop 2024-03-24T09:45:21.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.268+0000 7f09aea9b700 1 -- 172.21.15.22:0/584477525 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f09a8106a70 msgr2=0x7f09a819eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:21.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.268+0000 7f09aea9b700 1 --2- 172.21.15.22:0/584477525 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f09a8106a70 0x7f09a819eee0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f099400b4b0 tx=0x7f099400b870 comp rx=0 tx=0).stop 2024-03-24T09:45:21.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.268+0000 7f09aea9b700 1 -- 172.21.15.22:0/584477525 shutdown_connections 2024-03-24T09:45:21.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.268+0000 7f09aea9b700 1 --2- 172.21.15.22:0/584477525 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f099806cce0 0x7f099806f1a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:21.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.268+0000 7f09aea9b700 1 --2- 172.21.15.22:0/584477525 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09a8100a70 0x7f09a819e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:21.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.268+0000 7f09aea9b700 1 --2- 172.21.15.22:0/584477525 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f09a8106a70 0x7f09a819eee0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:21.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.268+0000 7f09aea9b700 1 -- 172.21.15.22:0/584477525 >> 172.21.15.22:0/584477525 conn(0x7f09a80fc5a0 msgr2=0x7f09a80732c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:45:21.269 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.268+0000 7f09aea9b700 1 -- 172.21.15.22:0/584477525 shutdown_connections 2024-03-24T09:45:21.270 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:21.268+0000 7f09aea9b700 1 -- 172.21.15.22:0/584477525 wait complete. 2024-03-24T09:45:21.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:21 smithi155 ceph-mon[37016]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:21.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:21 smithi022 ceph-mon[29512]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:21.901 INFO:tasks.cephadm:Setting up client nodes... 2024-03-24T09:45:21.902 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-24T09:45:22.111 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:45:22.422 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:22 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/584477525' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:45:22.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:22 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/584477525' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:45:23.456 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:23 smithi022 ceph-mon[29512]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:23.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:23 smithi155 ceph-mon[37016]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:23.678 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.676+0000 7f8418a35700 1 -- 172.21.15.22:0/2726550532 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8414103ea0 msgr2=0x7f8414106290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:23.678 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.676+0000 7f8418a35700 1 --2- 172.21.15.22:0/2726550532 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8414103ea0 0x7f8414106290 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f83fc009910 tx=0x7f83fc009c20 comp rx=0 tx=0).stop 2024-03-24T09:45:23.678 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.676+0000 7f8418a35700 1 -- 172.21.15.22:0/2726550532 shutdown_connections 2024-03-24T09:45:23.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.676+0000 7f8418a35700 1 --2- 172.21.15.22:0/2726550532 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8414103ea0 0x7f8414106290 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:23.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.676+0000 7f8418a35700 1 --2- 172.21.15.22:0/2726550532 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8414101570 0x7f8414103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:23.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.676+0000 7f8418a35700 1 -- 172.21.15.22:0/2726550532 >> 172.21.15.22:0/2726550532 conn(0x7f84140fafc0 msgr2=0x7f84140fd3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:45:23.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.676+0000 7f8418a35700 1 -- 172.21.15.22:0/2726550532 shutdown_connections 2024-03-24T09:45:23.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.676+0000 7f8418a35700 1 -- 172.21.15.22:0/2726550532 wait complete. 2024-03-24T09:45:23.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.676+0000 7f8418a35700 1 Processor -- start 2024-03-24T09:45:23.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.677+0000 7f8418a35700 1 -- start start 2024-03-24T09:45:23.679 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.677+0000 7f8418a35700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8414101570 0x7f8414196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:23.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.677+0000 7f8418a35700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8414103ea0 0x7f8414196ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:23.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.677+0000 7f8418a35700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84141970e0 con 0x7f8414103ea0 2024-03-24T09:45:23.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.677+0000 7f8418a35700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8414197220 con 0x7f8414101570 2024-03-24T09:45:23.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.677+0000 7f8411d9b700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8414103ea0 0x7f8414196ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:23.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.677+0000 7f8411d9b700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8414103ea0 0x7f8414196ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:45624/0 (socket says 172.21.15.22:45624) 2024-03-24T09:45:23.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.677+0000 7f8411d9b700 1 -- 172.21.15.22:0/4131032690 learned_addr learned my addr 172.21.15.22:0/4131032690 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:45:23.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.677+0000 7f8411d9b700 1 -- 172.21.15.22:0/4131032690 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8414101570 msgr2=0x7f8414196580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:23.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.677+0000 7f8411d9b700 1 --2- 172.21.15.22:0/4131032690 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8414101570 0x7f8414196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:23.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.677+0000 7f8411d9b700 1 -- 172.21.15.22:0/4131032690 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83fc0095f0 con 0x7f8414103ea0 2024-03-24T09:45:23.680 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.677+0000 7f8411d9b700 1 --2- 172.21.15.22:0/4131032690 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8414103ea0 0x7f8414196ac0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f83fc0092e0 tx=0x7f83fc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:23.681 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.677+0000 7f840b7fe700 1 -- 172.21.15.22:0/4131032690 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83fc01d070 con 0x7f8414103ea0 2024-03-24T09:45:23.681 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.677+0000 7f840b7fe700 1 -- 172.21.15.22:0/4131032690 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f83fc00f460 con 0x7f8414103ea0 2024-03-24T09:45:23.681 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.677+0000 7f8418a35700 1 -- 172.21.15.22:0/4131032690 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f841419bbb0 con 0x7f8414103ea0 2024-03-24T09:45:23.681 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.677+0000 7f840b7fe700 1 -- 172.21.15.22:0/4131032690 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83fc021640 con 0x7f8414103ea0 2024-03-24T09:45:23.681 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.677+0000 7f8418a35700 1 -- 172.21.15.22:0/4131032690 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f841419c120 con 0x7f8414103ea0 2024-03-24T09:45:23.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.678+0000 7f8418a35700 1 -- 172.21.15.22:0/4131032690 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f841404ea80 con 0x7f8414103ea0 2024-03-24T09:45:23.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.679+0000 7f840b7fe700 1 -- 172.21.15.22:0/4131032690 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f83fc00f5d0 con 0x7f8414103ea0 2024-03-24T09:45:23.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.679+0000 7f840b7fe700 1 --2- 172.21.15.22:0/4131032690 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f840006cce0 0x7f840006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:23.682 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.679+0000 7f840b7fe700 1 -- 172.21.15.22:0/4131032690 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f83fc08d280 con 0x7f8414103ea0 2024-03-24T09:45:23.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.679+0000 7f841259c700 1 --2- 172.21.15.22:0/4131032690 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f840006cce0 0x7f840006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:23.683 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.680+0000 7f841259c700 1 --2- 172.21.15.22:0/4131032690 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f840006cce0 0x7f840006f1a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f840c00ab90 tx=0x7f840c009250 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:23.684 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.684+0000 7f840b7fe700 1 -- 172.21.15.22:0/4131032690 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f83fc05b420 con 0x7f8414103ea0 2024-03-24T09:45:23.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.967+0000 7f8418a35700 1 -- 172.21.15.22:0/4131032690 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f841419c3d0 con 0x7f8414103ea0 2024-03-24T09:45:23.971 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.971+0000 7f840b7fe700 1 -- 172.21.15.22:0/4131032690 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) v1 ==== 170+0+59 (secure 0 0 0) 0x7f83fc05afb0 con 0x7f8414103ea0 2024-03-24T09:45:23.972 INFO:teuthology.orchestra.run.smithi022.stdout:[client.0] 2024-03-24T09:45:23.972 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQAz9v9lzXfHORAAy3f/o7mlmqlvjGB9iRroXg== 2024-03-24T09:45:23.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.973+0000 7f8418a35700 1 -- 172.21.15.22:0/4131032690 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f840006cce0 msgr2=0x7f840006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:23.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.973+0000 7f8418a35700 1 --2- 172.21.15.22:0/4131032690 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f840006cce0 0x7f840006f1a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f840c00ab90 tx=0x7f840c009250 comp rx=0 tx=0).stop 2024-03-24T09:45:23.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.973+0000 7f8418a35700 1 -- 172.21.15.22:0/4131032690 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8414103ea0 msgr2=0x7f8414196ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:23.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.973+0000 7f8418a35700 1 --2- 172.21.15.22:0/4131032690 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8414103ea0 0x7f8414196ac0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f83fc0092e0 tx=0x7f83fc00be20 comp rx=0 tx=0).stop 2024-03-24T09:45:23.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.973+0000 7f8418a35700 1 -- 172.21.15.22:0/4131032690 shutdown_connections 2024-03-24T09:45:23.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.973+0000 7f8418a35700 1 --2- 172.21.15.22:0/4131032690 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f840006cce0 0x7f840006f1a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:23.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.973+0000 7f8418a35700 1 --2- 172.21.15.22:0/4131032690 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8414101570 0x7f8414196580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:23.974 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.973+0000 7f8418a35700 1 --2- 172.21.15.22:0/4131032690 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8414103ea0 0x7f8414196ac0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:23.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.973+0000 7f8418a35700 1 -- 172.21.15.22:0/4131032690 >> 172.21.15.22:0/4131032690 conn(0x7f84140fafc0 msgr2=0x7f84140fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:45:23.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.973+0000 7f8418a35700 1 -- 172.21.15.22:0/4131032690 shutdown_connections 2024-03-24T09:45:23.975 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:23.973+0000 7f8418a35700 1 -- 172.21.15.22:0/4131032690 wait complete. 2024-03-24T09:45:24.242 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:24 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:45:24.242 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:24 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/4131032690' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T09:45:24.242 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:24 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/4131032690' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T09:45:24.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:24 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:45:24.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:24 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/4131032690' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T09:45:24.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:24 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/4131032690' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T09:45:24.639 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T09:45:24.639 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-24T09:45:24.639 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-24T09:45:24.683 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-24T09:45:24.901 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi155/config 2024-03-24T09:45:25.246 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:25 smithi155 ceph-mon[37016]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:25.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:25 smithi022 ceph-mon[29512]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:26.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.519+0000 7fd5d1fd4700 1 -- 172.21.15.155:0/3074154469 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5cc103f50 msgr2=0x7fd5cc106340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:26.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.519+0000 7fd5d1fd4700 1 --2- 172.21.15.155:0/3074154469 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5cc103f50 0x7fd5cc106340 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd5b4009910 tx=0x7fd5b4009c20 comp rx=0 tx=0).stop 2024-03-24T09:45:26.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.519+0000 7fd5d1fd4700 1 -- 172.21.15.155:0/3074154469 shutdown_connections 2024-03-24T09:45:26.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.519+0000 7fd5d1fd4700 1 --2- 172.21.15.155:0/3074154469 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5cc103f50 0x7fd5cc106340 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:26.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.519+0000 7fd5d1fd4700 1 --2- 172.21.15.155:0/3074154469 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd5cc101620 0x7fd5cc103a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:26.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.519+0000 7fd5d1fd4700 1 -- 172.21.15.155:0/3074154469 >> 172.21.15.155:0/3074154469 conn(0x7fd5cc0fafb0 msgr2=0x7fd5cc0fd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:45:26.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.520+0000 7fd5d1fd4700 1 -- 172.21.15.155:0/3074154469 shutdown_connections 2024-03-24T09:45:26.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.520+0000 7fd5d1fd4700 1 -- 172.21.15.155:0/3074154469 wait complete. 2024-03-24T09:45:26.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.520+0000 7fd5d1fd4700 1 Processor -- start 2024-03-24T09:45:26.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.520+0000 7fd5d1fd4700 1 -- start start 2024-03-24T09:45:26.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.520+0000 7fd5d1fd4700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5cc101620 0x7fd5cc196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:26.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.520+0000 7fd5d1fd4700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd5cc103f50 0x7fd5cc196ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:26.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.520+0000 7fd5d1fd4700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5cc1970e0 con 0x7fd5cc103f50 2024-03-24T09:45:26.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.520+0000 7fd5d1fd4700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5cc197220 con 0x7fd5cc101620 2024-03-24T09:45:26.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.520+0000 7fd5cb7fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5cc101620 0x7fd5cc196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:26.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.520+0000 7fd5cb7fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5cc101620 0x7fd5cc196580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:54036/0 (socket says 172.21.15.155:54036) 2024-03-24T09:45:26.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.520+0000 7fd5cb7fe700 1 -- 172.21.15.155:0/1485986647 learned_addr learned my addr 172.21.15.155:0/1485986647 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-24T09:45:26.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.520+0000 7fd5c3fff700 1 --2- 172.21.15.155:0/1485986647 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd5cc103f50 0x7fd5cc196ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:26.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.521+0000 7fd5cb7fe700 1 -- 172.21.15.155:0/1485986647 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd5cc103f50 msgr2=0x7fd5cc196ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:26.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.521+0000 7fd5cb7fe700 1 --2- 172.21.15.155:0/1485986647 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd5cc103f50 0x7fd5cc196ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:26.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.521+0000 7fd5cb7fe700 1 -- 172.21.15.155:0/1485986647 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b40095f0 con 0x7fd5cc101620 2024-03-24T09:45:26.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.521+0000 7fd5cb7fe700 1 --2- 172.21.15.155:0/1485986647 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5cc101620 0x7fd5cc196580 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc00d8d0 tx=0x7fd5bc00dbe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:26.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.521+0000 7fd5c97fa700 1 -- 172.21.15.155:0/1485986647 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5bc0098a0 con 0x7fd5cc101620 2024-03-24T09:45:26.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.521+0000 7fd5c97fa700 1 -- 172.21.15.155:0/1485986647 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd5bc010460 con 0x7fd5cc101620 2024-03-24T09:45:26.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.521+0000 7fd5d1fd4700 1 -- 172.21.15.155:0/1485986647 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5cc19bc30 con 0x7fd5cc101620 2024-03-24T09:45:26.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.521+0000 7fd5c97fa700 1 -- 172.21.15.155:0/1485986647 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5bc00f5f0 con 0x7fd5cc101620 2024-03-24T09:45:26.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.521+0000 7fd5d1fd4700 1 -- 172.21.15.155:0/1485986647 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5cc19c180 con 0x7fd5cc101620 2024-03-24T09:45:26.526 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.522+0000 7fd5d1fd4700 1 -- 172.21.15.155:0/1485986647 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5cc04ea80 con 0x7fd5cc101620 2024-03-24T09:45:26.527 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.523+0000 7fd5c97fa700 1 -- 172.21.15.155:0/1485986647 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7fd5bc009a00 con 0x7fd5cc101620 2024-03-24T09:45:26.527 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.523+0000 7fd5c97fa700 1 --2- 172.21.15.155:0/1485986647 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fd5b806cce0 0x7fd5b806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:26.527 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.523+0000 7fd5c97fa700 1 -- 172.21.15.155:0/1485986647 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fd5bc08bdc0 con 0x7fd5cc101620 2024-03-24T09:45:26.527 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.523+0000 7fd5c3fff700 1 --2- 172.21.15.155:0/1485986647 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fd5b806cce0 0x7fd5b806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:26.527 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.524+0000 7fd5c3fff700 1 --2- 172.21.15.155:0/1485986647 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fd5b806cce0 0x7fd5b806f1a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd5b40051c0 tx=0x7fd5b401a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:26.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.528+0000 7fd5c97fa700 1 -- 172.21.15.155:0/1485986647 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd5bc059f60 con 0x7fd5cc101620 2024-03-24T09:45:26.799 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.798+0000 7fd5d1fd4700 1 -- 172.21.15.155:0/1485986647 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fd5cc19c7b0 con 0x7fd5cc101620 2024-03-24T09:45:26.804 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.803+0000 7fd5c97fa700 1 -- 172.21.15.155:0/1485986647 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (secure 0 0 0) 0x7fd5bc059af0 con 0x7fd5cc101620 2024-03-24T09:45:26.804 INFO:teuthology.orchestra.run.smithi155.stdout:[client.1] 2024-03-24T09:45:26.804 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQA29v9lQBHFLxAAHXkNrOQ7aU848IMiWmniPg== 2024-03-24T09:45:26.806 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.805+0000 7fd5d1fd4700 1 -- 172.21.15.155:0/1485986647 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fd5b806cce0 msgr2=0x7fd5b806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:26.807 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.805+0000 7fd5d1fd4700 1 --2- 172.21.15.155:0/1485986647 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fd5b806cce0 0x7fd5b806f1a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd5b40051c0 tx=0x7fd5b401a040 comp rx=0 tx=0).stop 2024-03-24T09:45:26.807 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.805+0000 7fd5d1fd4700 1 -- 172.21.15.155:0/1485986647 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5cc101620 msgr2=0x7fd5cc196580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:26.807 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.805+0000 7fd5d1fd4700 1 --2- 172.21.15.155:0/1485986647 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5cc101620 0x7fd5cc196580 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc00d8d0 tx=0x7fd5bc00dbe0 comp rx=0 tx=0).stop 2024-03-24T09:45:26.807 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.805+0000 7fd5d1fd4700 1 -- 172.21.15.155:0/1485986647 shutdown_connections 2024-03-24T09:45:26.807 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.805+0000 7fd5d1fd4700 1 --2- 172.21.15.155:0/1485986647 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fd5b806cce0 0x7fd5b806f1a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:26.807 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.805+0000 7fd5d1fd4700 1 --2- 172.21.15.155:0/1485986647 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5cc101620 0x7fd5cc196580 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:26.807 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.805+0000 7fd5d1fd4700 1 --2- 172.21.15.155:0/1485986647 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd5cc103f50 0x7fd5cc196ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:26.807 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.805+0000 7fd5d1fd4700 1 -- 172.21.15.155:0/1485986647 >> 172.21.15.155:0/1485986647 conn(0x7fd5cc0fafb0 msgr2=0x7fd5cc0fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:45:26.807 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.805+0000 7fd5d1fd4700 1 -- 172.21.15.155:0/1485986647 shutdown_connections 2024-03-24T09:45:26.807 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:45:26.805+0000 7fd5d1fd4700 1 -- 172.21.15.155:0/1485986647 wait complete. 2024-03-24T09:45:27.419 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-24T09:45:27.419 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-24T09:45:27.419 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-24T09:45:27.462 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-24T09:45:27.463 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-24T09:45:27.463 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph mgr dump --format=json 2024-03-24T09:45:27.684 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:45:27.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:27 smithi155 ceph-mon[37016]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:27.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:27 smithi155 ceph-mon[37016]: from='client.? 172.21.15.155:0/1485986647' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T09:45:27.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:27 smithi155 ceph-mon[37016]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T09:45:27.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:27 smithi155 ceph-mon[37016]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T09:45:27.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:27 smithi022 ceph-mon[29512]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:27.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:27 smithi022 ceph-mon[29512]: from='client.? 172.21.15.155:0/1485986647' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T09:45:27.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:27 smithi022 ceph-mon[29512]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T09:45:27.783 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:27 smithi022 ceph-mon[29512]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T09:45:29.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.391+0000 7f712ad77700 1 -- 172.21.15.22:0/1379446499 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7124100370 msgr2=0x7f7124104840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:29.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.391+0000 7f712ad77700 1 --2- 172.21.15.22:0/1379446499 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7124100370 0x7f7124104840 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7110009910 tx=0x7f7110009c20 comp rx=0 tx=0).stop 2024-03-24T09:45:29.393 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.391+0000 7f712ad77700 1 -- 172.21.15.22:0/1379446499 shutdown_connections 2024-03-24T09:45:29.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.391+0000 7f712ad77700 1 --2- 172.21.15.22:0/1379446499 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7124100370 0x7f7124104840 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:29.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.391+0000 7f712ad77700 1 --2- 172.21.15.22:0/1379446499 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71240ffa50 0x7f71240ffe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:29.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.391+0000 7f712ad77700 1 -- 172.21.15.22:0/1379446499 >> 172.21.15.22:0/1379446499 conn(0x7f71240fb4b0 msgr2=0x7f71240fd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:45:29.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.391+0000 7f712ad77700 1 -- 172.21.15.22:0/1379446499 shutdown_connections 2024-03-24T09:45:29.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.391+0000 7f712ad77700 1 -- 172.21.15.22:0/1379446499 wait complete. 2024-03-24T09:45:29.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.391+0000 7f712ad77700 1 Processor -- start 2024-03-24T09:45:29.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.391+0000 7f712ad77700 1 -- start start 2024-03-24T09:45:29.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.391+0000 7f712ad77700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f71240ffa50 0x7f712419d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:29.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.391+0000 7f712ad77700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7124100370 0x7f712419d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:29.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.391+0000 7f712ad77700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712419dd30 con 0x7f71240ffa50 2024-03-24T09:45:29.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.391+0000 7f712ad77700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71241a1a00 con 0x7f7124100370 2024-03-24T09:45:29.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.392+0000 7f7128b13700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f71240ffa50 0x7f712419d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:29.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.392+0000 7f7128b13700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f71240ffa50 0x7f712419d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:46160/0 (socket says 172.21.15.22:46160) 2024-03-24T09:45:29.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.392+0000 7f7128b13700 1 -- 172.21.15.22:0/2086016928 learned_addr learned my addr 172.21.15.22:0/2086016928 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:45:29.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.392+0000 7f7128b13700 1 -- 172.21.15.22:0/2086016928 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7124100370 msgr2=0x7f712419d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:29.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.392+0000 7f7128b13700 1 --2- 172.21.15.22:0/2086016928 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7124100370 0x7f712419d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:29.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.392+0000 7f7128b13700 1 -- 172.21.15.22:0/2086016928 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71100095f0 con 0x7f71240ffa50 2024-03-24T09:45:29.395 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.392+0000 7f7128b13700 1 --2- 172.21.15.22:0/2086016928 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f71240ffa50 0x7f712419d110 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f711800b4b0 tx=0x7f711800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:29.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.392+0000 7f7121ffb700 1 -- 172.21.15.22:0/2086016928 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7118011800 con 0x7f71240ffa50 2024-03-24T09:45:29.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.392+0000 7f7121ffb700 1 -- 172.21.15.22:0/2086016928 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7118011e20 con 0x7f71240ffa50 2024-03-24T09:45:29.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.392+0000 7f712ad77700 1 -- 172.21.15.22:0/2086016928 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71241a1ce0 con 0x7f71240ffa50 2024-03-24T09:45:29.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.392+0000 7f7121ffb700 1 -- 172.21.15.22:0/2086016928 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f711800f2e0 con 0x7f71240ffa50 2024-03-24T09:45:29.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.392+0000 7f712ad77700 1 -- 172.21.15.22:0/2086016928 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71241a2230 con 0x7f71240ffa50 2024-03-24T09:45:29.397 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.393+0000 7f712ad77700 1 -- 172.21.15.22:0/2086016928 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71240698b0 con 0x7f71240ffa50 2024-03-24T09:45:29.397 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.394+0000 7f7121ffb700 1 -- 172.21.15.22:0/2086016928 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f7118010380 con 0x7f71240ffa50 2024-03-24T09:45:29.397 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.394+0000 7f7121ffb700 1 --2- 172.21.15.22:0/2086016928 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f711406cce0 0x7f711406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:29.397 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.394+0000 7f7123fff700 1 --2- 172.21.15.22:0/2086016928 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f711406cce0 0x7f711406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:29.397 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.394+0000 7f7121ffb700 1 -- 172.21.15.22:0/2086016928 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f711808bae0 con 0x7f71240ffa50 2024-03-24T09:45:29.398 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.395+0000 7f7123fff700 1 --2- 172.21.15.22:0/2086016928 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f711406cce0 0x7f711406f1a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7110009fd0 tx=0x7f711000b580 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:29.399 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.399+0000 7f7121ffb700 1 -- 172.21.15.22:0/2086016928 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f71180561f0 con 0x7f71240ffa50 2024-03-24T09:45:29.658 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.657+0000 7f712ad77700 1 -- 172.21.15.22:0/2086016928 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f71241a24b0 con 0x7f71240ffa50 2024-03-24T09:45:29.661 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.661+0000 7f7121ffb700 1 -- 172.21.15.22:0/2086016928 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+173998 (secure 0 0 0) 0x7f7118059810 con 0x7f71240ffa50 2024-03-24T09:45:29.665 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:45:29.667 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.663+0000 7f712ad77700 1 -- 172.21.15.22:0/2086016928 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f711406cce0 msgr2=0x7f711406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:29.667 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.663+0000 7f712ad77700 1 --2- 172.21.15.22:0/2086016928 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f711406cce0 0x7f711406f1a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7110009fd0 tx=0x7f711000b580 comp rx=0 tx=0).stop 2024-03-24T09:45:29.667 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.663+0000 7f712ad77700 1 -- 172.21.15.22:0/2086016928 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f71240ffa50 msgr2=0x7f712419d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:29.667 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.663+0000 7f712ad77700 1 --2- 172.21.15.22:0/2086016928 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f71240ffa50 0x7f712419d110 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f711800b4b0 tx=0x7f711800b870 comp rx=0 tx=0).stop 2024-03-24T09:45:29.667 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.663+0000 7f712ad77700 1 -- 172.21.15.22:0/2086016928 shutdown_connections 2024-03-24T09:45:29.667 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.663+0000 7f712ad77700 1 --2- 172.21.15.22:0/2086016928 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f711406cce0 0x7f711406f1a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:29.668 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.663+0000 7f712ad77700 1 --2- 172.21.15.22:0/2086016928 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f71240ffa50 0x7f712419d110 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:29.668 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.663+0000 7f712ad77700 1 --2- 172.21.15.22:0/2086016928 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7124100370 0x7f712419d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:29.668 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.663+0000 7f712ad77700 1 -- 172.21.15.22:0/2086016928 >> 172.21.15.22:0/2086016928 conn(0x7f71240fb4b0 msgr2=0x7f712410e620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:45:29.668 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.663+0000 7f712ad77700 1 -- 172.21.15.22:0/2086016928 shutdown_connections 2024-03-24T09:45:29.668 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:29.663+0000 7f712ad77700 1 -- 172.21.15.22:0/2086016928 wait complete. 2024-03-24T09:45:29.854 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:29 smithi022 ceph-mon[29512]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:29.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:29 smithi155 ceph-mon[37016]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:30.239 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":19,"active_gid":14215,"active_name":"smithi022.emgmxz","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6800","nonce":773863266},{"type":"v1","addr":"172.21.15.22:6801","nonce":773863266}]},"active_addr":"172.21.15.22:6801/773863266","active_change":"2024-03-24T09:41:38.451450+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":14240,"name":"smithi155.itwzew","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.22:8443/","prometheus":"http://172.21.15.22:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.22:0","nonce":1432458684}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.22:0","nonce":4000690486}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.22:0","nonce":42033119}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.22:0","nonce":2833056868}]}]} 2024-03-24T09:45:30.243 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-24T09:45:30.244 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-24T09:45:30.244 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T09:45:30.457 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:45:30.867 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:30 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/2086016928' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T09:45:30.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:30 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/2086016928' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T09:45:31.761 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:31 smithi022 ceph-mon[29512]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:31.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:31 smithi155 ceph-mon[37016]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:32.054 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.000+0000 7f539d93f700 1 -- 172.21.15.22:0/91513613 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5398105950 msgr2=0x7f5398105d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:32.054 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.000+0000 7f539d93f700 1 --2- 172.21.15.22:0/91513613 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5398105950 0x7f5398105d30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f538800b0a0 tx=0x7f538800b3b0 comp rx=0 tx=0).stop 2024-03-24T09:45:32.054 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.000+0000 7f539d93f700 1 -- 172.21.15.22:0/91513613 shutdown_connections 2024-03-24T09:45:32.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.000+0000 7f539d93f700 1 --2- 172.21.15.22:0/91513613 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f53980ff950 0x7f53980ffdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:32.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.000+0000 7f539d93f700 1 --2- 172.21.15.22:0/91513613 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5398105950 0x7f5398105d30 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:32.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.000+0000 7f539d93f700 1 -- 172.21.15.22:0/91513613 >> 172.21.15.22:0/91513613 conn(0x7f5398076640 msgr2=0x7f53980fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:45:32.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.000+0000 7f539d93f700 1 -- 172.21.15.22:0/91513613 shutdown_connections 2024-03-24T09:45:32.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.000+0000 7f539d93f700 1 -- 172.21.15.22:0/91513613 wait complete. 2024-03-24T09:45:32.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.000+0000 7f539d93f700 1 Processor -- start 2024-03-24T09:45:32.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.001+0000 7f539d93f700 1 -- start start 2024-03-24T09:45:32.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.001+0000 7f539d93f700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f53980ff950 0x7f5398197770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:32.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.001+0000 7f539d93f700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5398105950 0x7f5398197cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:32.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.001+0000 7f539d93f700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5398198280 con 0x7f53980ff950 2024-03-24T09:45:32.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.001+0000 7f539d93f700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53981983c0 con 0x7f5398105950 2024-03-24T09:45:32.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.001+0000 7f5396ffd700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f53980ff950 0x7f5398197770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:32.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.001+0000 7f5396ffd700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f53980ff950 0x7f5398197770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:46168/0 (socket says 172.21.15.22:46168) 2024-03-24T09:45:32.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.001+0000 7f5396ffd700 1 -- 172.21.15.22:0/457902344 learned_addr learned my addr 172.21.15.22:0/457902344 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:45:32.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.001+0000 7f53967fc700 1 --2- 172.21.15.22:0/457902344 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5398105950 0x7f5398197cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:32.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.001+0000 7f5396ffd700 1 -- 172.21.15.22:0/457902344 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5398105950 msgr2=0x7f5398197cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:32.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.001+0000 7f5396ffd700 1 --2- 172.21.15.22:0/457902344 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5398105950 0x7f5398197cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:32.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.001+0000 7f5396ffd700 1 -- 172.21.15.22:0/457902344 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5388009d00 con 0x7f53980ff950 2024-03-24T09:45:32.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.001+0000 7f53967fc700 1 --2- 172.21.15.22:0/457902344 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5398105950 0x7f5398197cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:45:32.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.001+0000 7f5396ffd700 1 --2- 172.21.15.22:0/457902344 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f53980ff950 0x7f5398197770 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5388000f80 tx=0x7f5388007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:32.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.002+0000 7f539c93d700 1 -- 172.21.15.22:0/457902344 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f538800e040 con 0x7f53980ff950 2024-03-24T09:45:32.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.002+0000 7f539c93d700 1 -- 172.21.15.22:0/457902344 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5388003c10 con 0x7f53980ff950 2024-03-24T09:45:32.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.002+0000 7f539d93f700 1 -- 172.21.15.22:0/457902344 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53981a07e0 con 0x7f53980ff950 2024-03-24T09:45:32.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.002+0000 7f539c93d700 1 -- 172.21.15.22:0/457902344 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f538801d6b0 con 0x7f53980ff950 2024-03-24T09:45:32.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.002+0000 7f539d93f700 1 -- 172.21.15.22:0/457902344 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53981a0b90 con 0x7f53980ff950 2024-03-24T09:45:32.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.003+0000 7f539d93f700 1 -- 172.21.15.22:0/457902344 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5398107f50 con 0x7f53980ff950 2024-03-24T09:45:32.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.003+0000 7f539c93d700 1 -- 172.21.15.22:0/457902344 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f5388004040 con 0x7f53980ff950 2024-03-24T09:45:32.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.004+0000 7f539c93d700 1 --2- 172.21.15.22:0/457902344 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f538406cce0 0x7f538406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:32.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.004+0000 7f53967fc700 1 --2- 172.21.15.22:0/457902344 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f538406cce0 0x7f538406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:32.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.004+0000 7f539c93d700 1 -- 172.21.15.22:0/457902344 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f538808d790 con 0x7f53980ff950 2024-03-24T09:45:32.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.004+0000 7f53967fc700 1 --2- 172.21.15.22:0/457902344 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f538406cce0 0x7f538406f1a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5380005950 tx=0x7f53800058e0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:32.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.008+0000 7f539c93d700 1 -- 172.21.15.22:0/457902344 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5388057e00 con 0x7f53980ff950 2024-03-24T09:45:32.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.211+0000 7f539d93f700 1 -- 172.21.15.22:0/457902344 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5398068a80 con 0x7f53980ff950 2024-03-24T09:45:32.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.211+0000 7f539c93d700 1 -- 172.21.15.22:0/457902344 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11348 (secure 0 0 0) 0x7f538805b420 con 0x7f53980ff950 2024-03-24T09:45:32.212 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:45:32.212 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":40,"fsid":"45356448-e9c2-11ee-95cd-87774f69a715","created":"2024-03-24T09:39:25.502234+0000","modified":"2024-03-24T09:45:13.574726+0000","last_up_change":"2024-03-24T09:45:12.567478+0000","last_in_change":"2024-03-24T09:44:53.111067+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T09:43:56.788035+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ba411044-c790-44b5-881e-4e86675a868b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":3431970421},{"type":"v1","addr":"172.21.15.22:6803","nonce":3431970421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":3431970421},{"type":"v1","addr":"172.21.15.22:6805","nonce":3431970421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":3431970421},{"type":"v1","addr":"172.21.15.22:6809","nonce":3431970421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":3431970421},{"type":"v1","addr":"172.21.15.22:6807","nonce":3431970421}]},"public_addr":"172.21.15.22:6803/3431970421","cluster_addr":"172.21.15.22:6805/3431970421","heartbeat_back_addr":"172.21.15.22:6809/3431970421","heartbeat_front_addr":"172.21.15.22:6807/3431970421","state":["exists","up"]},{"osd":1,"uuid":"2bc9ee9b-4d76-4df5-bc42-92f243ba6c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":1675089147},{"type":"v1","addr":"172.21.15.22:6811","nonce":1675089147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":1675089147},{"type":"v1","addr":"172.21.15.22:6813","nonce":1675089147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":1675089147},{"type":"v1","addr":"172.21.15.22:6817","nonce":1675089147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":1675089147},{"type":"v1","addr":"172.21.15.22:6815","nonce":1675089147}]},"public_addr":"172.21.15.22:6811/1675089147","cluster_addr":"172.21.15.22:6813/1675089147","heartbeat_back_addr":"172.21.15.22:6817/1675089147","heartbeat_front_addr":"172.21.15.22:6815/1675089147","state":["exists","up"]},{"osd":2,"uuid":"1e0c3afd-b47b-4de8-ab3c-e25770557c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":1538987315},{"type":"v1","addr":"172.21.15.22:6819","nonce":1538987315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":1538987315},{"type":"v1","addr":"172.21.15.22:6821","nonce":1538987315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":1538987315},{"type":"v1","addr":"172.21.15.22:6825","nonce":1538987315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":1538987315},{"type":"v1","addr":"172.21.15.22:6823","nonce":1538987315}]},"public_addr":"172.21.15.22:6819/1538987315","cluster_addr":"172.21.15.22:6821/1538987315","heartbeat_back_addr":"172.21.15.22:6825/1538987315","heartbeat_front_addr":"172.21.15.22:6823/1538987315","state":["exists","up"]},{"osd":3,"uuid":"56f51449-b4b8-4874-9469-497ce984a5f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":2405415706},{"type":"v1","addr":"172.21.15.155:6801","nonce":2405415706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":2405415706},{"type":"v1","addr":"172.21.15.155:6803","nonce":2405415706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":2405415706},{"type":"v1","addr":"172.21.15.155:6807","nonce":2405415706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":2405415706},{"type":"v1","addr":"172.21.15.155:6805","nonce":2405415706}]},"public_addr":"172.21.15.155:6801/2405415706","cluster_addr":"172.21.15.155:6803/2405415706","heartbeat_back_addr":"172.21.15.155:6807/2405415706","heartbeat_front_addr":"172.21.15.155:6805/2405415706","state":["exists","up"]},{"osd":4,"uuid":"1af86297-1514-495a-8d74-6d0237f82316","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":2965421758},{"type":"v1","addr":"172.21.15.155:6809","nonce":2965421758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":2965421758},{"type":"v1","addr":"172.21.15.155:6811","nonce":2965421758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":2965421758},{"type":"v1","addr":"172.21.15.155:6815","nonce":2965421758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":2965421758},{"type":"v1","addr":"172.21.15.155:6813","nonce":2965421758}]},"public_addr":"172.21.15.155:6809/2965421758","cluster_addr":"172.21.15.155:6811/2965421758","heartbeat_back_addr":"172.21.15.155:6815/2965421758","heartbeat_front_addr":"172.21.15.155:6813/2965421758","state":["exists","up"]},{"osd":5,"uuid":"aa6a8e29-123e-4b1f-8bed-8b207cfadce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3664800012},{"type":"v1","addr":"172.21.15.155:6817","nonce":3664800012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3664800012},{"type":"v1","addr":"172.21.15.155:6819","nonce":3664800012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3664800012},{"type":"v1","addr":"172.21.15.155:6823","nonce":3664800012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3664800012},{"type":"v1","addr":"172.21.15.155:6821","nonce":3664800012}]},"public_addr":"172.21.15.155:6817/3664800012","cluster_addr":"172.21.15.155:6819/3664800012","heartbeat_back_addr":"172.21.15.155:6823/3664800012","heartbeat_front_addr":"172.21.15.155:6821/3664800012","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:42:57.929167+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:43:25.482421+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:43:54.589274+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:44:18.882506+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:44:44.731273+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:45:10.622995+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.22:0/632176108":"2024-03-25T09:41:38.451303+0000","172.21.15.22:6801/2979399337":"2024-03-25T09:41:38.451303+0000","172.21.15.22:0/1096383264":"2024-03-25T09:41:38.451303+0000","172.21.15.22:0/774594813":"2024-03-25T09:40:30.945227+0000","172.21.15.22:6801/3036078499":"2024-03-25T09:40:30.945227+0000","172.21.15.22:6800/3036078499":"2024-03-25T09:40:30.945227+0000","172.21.15.22:0/1182234699":"2024-03-25T09:40:30.945227+0000","172.21.15.22:0/3835020536":"2024-03-25T09:41:38.451303+0000","172.21.15.22:0/2826976179":"2024-03-25T09:39:58.019245+0000","172.21.15.22:0/363626647":"2024-03-25T09:39:58.019245+0000","172.21.15.22:6801/2423771373":"2024-03-25T09:39:58.019245+0000","172.21.15.22:6800/2979399337":"2024-03-25T09:41:38.451303+0000","172.21.15.22:0/2928401592":"2024-03-25T09:39:58.019245+0000","172.21.15.22:6800/2423771373":"2024-03-25T09:39:58.019245+0000","172.21.15.22:0/1186018607":"2024-03-25T09:40:30.945227+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T09:45:32.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.213+0000 7f539d93f700 1 -- 172.21.15.22:0/457902344 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f538406cce0 msgr2=0x7f538406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:32.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.213+0000 7f539d93f700 1 --2- 172.21.15.22:0/457902344 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f538406cce0 0x7f538406f1a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5380005950 tx=0x7f53800058e0 comp rx=0 tx=0).stop 2024-03-24T09:45:32.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.213+0000 7f539d93f700 1 -- 172.21.15.22:0/457902344 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f53980ff950 msgr2=0x7f5398197770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:32.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.213+0000 7f539d93f700 1 --2- 172.21.15.22:0/457902344 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f53980ff950 0x7f5398197770 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5388000f80 tx=0x7f5388007bd0 comp rx=0 tx=0).stop 2024-03-24T09:45:32.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.213+0000 7f539d93f700 1 -- 172.21.15.22:0/457902344 shutdown_connections 2024-03-24T09:45:32.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.214+0000 7f539d93f700 1 --2- 172.21.15.22:0/457902344 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f538406cce0 0x7f538406f1a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:32.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.214+0000 7f539d93f700 1 --2- 172.21.15.22:0/457902344 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f53980ff950 0x7f5398197770 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:32.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.214+0000 7f539d93f700 1 --2- 172.21.15.22:0/457902344 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5398105950 0x7f5398197cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:32.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.214+0000 7f539d93f700 1 -- 172.21.15.22:0/457902344 >> 172.21.15.22:0/457902344 conn(0x7f5398076640 msgr2=0x7f53980fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:45:32.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.214+0000 7f539d93f700 1 -- 172.21.15.22:0/457902344 shutdown_connections 2024-03-24T09:45:32.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:32.214+0000 7f539d93f700 1 -- 172.21.15.22:0/457902344 wait complete. 2024-03-24T09:45:32.911 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:32 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/457902344' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:45:32.935 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-24T09:45:32.935 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T09:45:32.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:32 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/457902344' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:45:33.149 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:45:33.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:33 smithi155 ceph-mon[37016]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:34.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:33 smithi022 ceph-mon[29512]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:34.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:34.823+0000 7f9657a3d700 1 -- 172.21.15.22:0/2328775152 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9650106a90 msgr2=0x7f9650106e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:34.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:34.823+0000 7f9657a3d700 1 --2- 172.21.15.22:0/2328775152 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9650106a90 0x7f9650106e70 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f964000b0a0 tx=0x7f964000b3b0 comp rx=0 tx=0).stop 2024-03-24T09:45:34.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:34.823+0000 7f9657a3d700 1 -- 172.21.15.22:0/2328775152 shutdown_connections 2024-03-24T09:45:34.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:34.823+0000 7f9657a3d700 1 --2- 172.21.15.22:0/2328775152 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9650100a90 0x7f9650100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:34.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:34.823+0000 7f9657a3d700 1 --2- 172.21.15.22:0/2328775152 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9650106a90 0x7f9650106e70 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:34.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:34.823+0000 7f9657a3d700 1 -- 172.21.15.22:0/2328775152 >> 172.21.15.22:0/2328775152 conn(0x7f96500fc580 msgr2=0x7f96500fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:45:34.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:34.823+0000 7f9657a3d700 1 -- 172.21.15.22:0/2328775152 shutdown_connections 2024-03-24T09:45:34.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:34.823+0000 7f9657a3d700 1 -- 172.21.15.22:0/2328775152 wait complete. 2024-03-24T09:45:34.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:34.824+0000 7f9657a3d700 1 Processor -- start 2024-03-24T09:45:34.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:34.824+0000 7f9657a3d700 1 -- start start 2024-03-24T09:45:34.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:34.824+0000 7f9657a3d700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9650100a90 0x7f96500730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:34.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:34.824+0000 7f9657a3d700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9650106a90 0x7f96500735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:34.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:34.824+0000 7f9657a3d700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9650073b30 con 0x7f9650106a90 2024-03-24T09:45:34.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:34.824+0000 7f9657a3d700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9650073c70 con 0x7f9650100a90 2024-03-24T09:45:34.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:34.824+0000 7f9654fd8700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9650106a90 0x7f96500735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:34.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:34.824+0000 7f9654fd8700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9650106a90 0x7f96500735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:46174/0 (socket says 172.21.15.22:46174) 2024-03-24T09:45:34.839 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:34.824+0000 7f9654fd8700 1 -- 172.21.15.22:0/3774224475 learned_addr learned my addr 172.21.15.22:0/3774224475 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:45:34.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:34.824+0000 7f96557d9700 1 --2- 172.21.15.22:0/3774224475 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9650100a90 0x7f96500730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:34.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:34.824+0000 7f9654fd8700 1 -- 172.21.15.22:0/3774224475 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9650100a90 msgr2=0x7f96500730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:34.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:34.824+0000 7f9654fd8700 1 --2- 172.21.15.22:0/3774224475 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9650100a90 0x7f96500730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:34.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:34.824+0000 7f9654fd8700 1 -- 172.21.15.22:0/3774224475 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9640009d00 con 0x7f9650106a90 2024-03-24T09:45:34.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:34.824+0000 7f9654fd8700 1 --2- 172.21.15.22:0/3774224475 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9650106a90 0x7f96500735f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f964400b4b0 tx=0x7f964400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:34.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:34.824+0000 7f964e7fc700 1 -- 172.21.15.22:0/3774224475 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9644011800 con 0x7f9650106a90 2024-03-24T09:45:34.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:34.824+0000 7f964e7fc700 1 -- 172.21.15.22:0/3774224475 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9644011e20 con 0x7f9650106a90 2024-03-24T09:45:34.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:34.825+0000 7f9657a3d700 1 -- 172.21.15.22:0/3774224475 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96501a2c70 con 0x7f9650106a90 2024-03-24T09:45:34.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:34.825+0000 7f964e7fc700 1 -- 172.21.15.22:0/3774224475 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f964400f2e0 con 0x7f9650106a90 2024-03-24T09:45:34.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:34.825+0000 7f9657a3d700 1 -- 172.21.15.22:0/3774224475 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96501a2fd0 con 0x7f9650106a90 2024-03-24T09:45:34.840 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:34.826+0000 7f9657a3d700 1 -- 172.21.15.22:0/3774224475 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9650109090 con 0x7f9650106a90 2024-03-24T09:45:34.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:34.826+0000 7f964e7fc700 1 -- 172.21.15.22:0/3774224475 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f9644010380 con 0x7f9650106a90 2024-03-24T09:45:34.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:34.826+0000 7f964e7fc700 1 --2- 172.21.15.22:0/3774224475 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f963806cce0 0x7f963806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:34.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:34.827+0000 7f96557d9700 1 --2- 172.21.15.22:0/3774224475 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f963806cce0 0x7f963806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:34.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:34.827+0000 7f964e7fc700 1 -- 172.21.15.22:0/3774224475 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f964408bae0 con 0x7f9650106a90 2024-03-24T09:45:34.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:34.827+0000 7f96557d9700 1 --2- 172.21.15.22:0/3774224475 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f963806cce0 0x7f963806f1a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f964000b830 tx=0x7f9640006000 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:34.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:34.831+0000 7f964e7fc700 1 -- 172.21.15.22:0/3774224475 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f96440561f0 con 0x7f9650106a90 2024-03-24T09:45:35.030 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:35.030+0000 7f9657a3d700 1 -- 172.21.15.22:0/3774224475 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96501021d0 con 0x7f9650106a90 2024-03-24T09:45:35.031 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:35.031+0000 7f964e7fc700 1 -- 172.21.15.22:0/3774224475 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11348 (secure 0 0 0) 0x7f9644059810 con 0x7f9650106a90 2024-03-24T09:45:35.032 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:45:35.032 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":40,"fsid":"45356448-e9c2-11ee-95cd-87774f69a715","created":"2024-03-24T09:39:25.502234+0000","modified":"2024-03-24T09:45:13.574726+0000","last_up_change":"2024-03-24T09:45:12.567478+0000","last_in_change":"2024-03-24T09:44:53.111067+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T09:43:56.788035+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ba411044-c790-44b5-881e-4e86675a868b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":3431970421},{"type":"v1","addr":"172.21.15.22:6803","nonce":3431970421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":3431970421},{"type":"v1","addr":"172.21.15.22:6805","nonce":3431970421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":3431970421},{"type":"v1","addr":"172.21.15.22:6809","nonce":3431970421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":3431970421},{"type":"v1","addr":"172.21.15.22:6807","nonce":3431970421}]},"public_addr":"172.21.15.22:6803/3431970421","cluster_addr":"172.21.15.22:6805/3431970421","heartbeat_back_addr":"172.21.15.22:6809/3431970421","heartbeat_front_addr":"172.21.15.22:6807/3431970421","state":["exists","up"]},{"osd":1,"uuid":"2bc9ee9b-4d76-4df5-bc42-92f243ba6c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":1675089147},{"type":"v1","addr":"172.21.15.22:6811","nonce":1675089147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":1675089147},{"type":"v1","addr":"172.21.15.22:6813","nonce":1675089147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":1675089147},{"type":"v1","addr":"172.21.15.22:6817","nonce":1675089147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":1675089147},{"type":"v1","addr":"172.21.15.22:6815","nonce":1675089147}]},"public_addr":"172.21.15.22:6811/1675089147","cluster_addr":"172.21.15.22:6813/1675089147","heartbeat_back_addr":"172.21.15.22:6817/1675089147","heartbeat_front_addr":"172.21.15.22:6815/1675089147","state":["exists","up"]},{"osd":2,"uuid":"1e0c3afd-b47b-4de8-ab3c-e25770557c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":1538987315},{"type":"v1","addr":"172.21.15.22:6819","nonce":1538987315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":1538987315},{"type":"v1","addr":"172.21.15.22:6821","nonce":1538987315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":1538987315},{"type":"v1","addr":"172.21.15.22:6825","nonce":1538987315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":1538987315},{"type":"v1","addr":"172.21.15.22:6823","nonce":1538987315}]},"public_addr":"172.21.15.22:6819/1538987315","cluster_addr":"172.21.15.22:6821/1538987315","heartbeat_back_addr":"172.21.15.22:6825/1538987315","heartbeat_front_addr":"172.21.15.22:6823/1538987315","state":["exists","up"]},{"osd":3,"uuid":"56f51449-b4b8-4874-9469-497ce984a5f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":2405415706},{"type":"v1","addr":"172.21.15.155:6801","nonce":2405415706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":2405415706},{"type":"v1","addr":"172.21.15.155:6803","nonce":2405415706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":2405415706},{"type":"v1","addr":"172.21.15.155:6807","nonce":2405415706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":2405415706},{"type":"v1","addr":"172.21.15.155:6805","nonce":2405415706}]},"public_addr":"172.21.15.155:6801/2405415706","cluster_addr":"172.21.15.155:6803/2405415706","heartbeat_back_addr":"172.21.15.155:6807/2405415706","heartbeat_front_addr":"172.21.15.155:6805/2405415706","state":["exists","up"]},{"osd":4,"uuid":"1af86297-1514-495a-8d74-6d0237f82316","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":2965421758},{"type":"v1","addr":"172.21.15.155:6809","nonce":2965421758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":2965421758},{"type":"v1","addr":"172.21.15.155:6811","nonce":2965421758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":2965421758},{"type":"v1","addr":"172.21.15.155:6815","nonce":2965421758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":2965421758},{"type":"v1","addr":"172.21.15.155:6813","nonce":2965421758}]},"public_addr":"172.21.15.155:6809/2965421758","cluster_addr":"172.21.15.155:6811/2965421758","heartbeat_back_addr":"172.21.15.155:6815/2965421758","heartbeat_front_addr":"172.21.15.155:6813/2965421758","state":["exists","up"]},{"osd":5,"uuid":"aa6a8e29-123e-4b1f-8bed-8b207cfadce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3664800012},{"type":"v1","addr":"172.21.15.155:6817","nonce":3664800012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3664800012},{"type":"v1","addr":"172.21.15.155:6819","nonce":3664800012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3664800012},{"type":"v1","addr":"172.21.15.155:6823","nonce":3664800012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3664800012},{"type":"v1","addr":"172.21.15.155:6821","nonce":3664800012}]},"public_addr":"172.21.15.155:6817/3664800012","cluster_addr":"172.21.15.155:6819/3664800012","heartbeat_back_addr":"172.21.15.155:6823/3664800012","heartbeat_front_addr":"172.21.15.155:6821/3664800012","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:42:57.929167+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:43:25.482421+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:43:54.589274+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:44:18.882506+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:44:44.731273+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:45:10.622995+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.22:0/632176108":"2024-03-25T09:41:38.451303+0000","172.21.15.22:6801/2979399337":"2024-03-25T09:41:38.451303+0000","172.21.15.22:0/1096383264":"2024-03-25T09:41:38.451303+0000","172.21.15.22:0/774594813":"2024-03-25T09:40:30.945227+0000","172.21.15.22:6801/3036078499":"2024-03-25T09:40:30.945227+0000","172.21.15.22:6800/3036078499":"2024-03-25T09:40:30.945227+0000","172.21.15.22:0/1182234699":"2024-03-25T09:40:30.945227+0000","172.21.15.22:0/3835020536":"2024-03-25T09:41:38.451303+0000","172.21.15.22:0/2826976179":"2024-03-25T09:39:58.019245+0000","172.21.15.22:0/363626647":"2024-03-25T09:39:58.019245+0000","172.21.15.22:6801/2423771373":"2024-03-25T09:39:58.019245+0000","172.21.15.22:6800/2979399337":"2024-03-25T09:41:38.451303+0000","172.21.15.22:0/2928401592":"2024-03-25T09:39:58.019245+0000","172.21.15.22:6800/2423771373":"2024-03-25T09:39:58.019245+0000","172.21.15.22:0/1186018607":"2024-03-25T09:40:30.945227+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T09:45:35.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:35.033+0000 7f9657a3d700 1 -- 172.21.15.22:0/3774224475 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f963806cce0 msgr2=0x7f963806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:35.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:35.033+0000 7f9657a3d700 1 --2- 172.21.15.22:0/3774224475 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f963806cce0 0x7f963806f1a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f964000b830 tx=0x7f9640006000 comp rx=0 tx=0).stop 2024-03-24T09:45:35.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:35.033+0000 7f9657a3d700 1 -- 172.21.15.22:0/3774224475 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9650106a90 msgr2=0x7f96500735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:35.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:35.033+0000 7f9657a3d700 1 --2- 172.21.15.22:0/3774224475 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9650106a90 0x7f96500735f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f964400b4b0 tx=0x7f964400b870 comp rx=0 tx=0).stop 2024-03-24T09:45:35.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:35.033+0000 7f9657a3d700 1 -- 172.21.15.22:0/3774224475 shutdown_connections 2024-03-24T09:45:35.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:35.033+0000 7f9657a3d700 1 --2- 172.21.15.22:0/3774224475 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f963806cce0 0x7f963806f1a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:35.034 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:35.033+0000 7f9657a3d700 1 --2- 172.21.15.22:0/3774224475 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9650100a90 0x7f96500730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:35.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:35.033+0000 7f9657a3d700 1 --2- 172.21.15.22:0/3774224475 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9650106a90 0x7f96500735f0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:35.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:35.033+0000 7f9657a3d700 1 -- 172.21.15.22:0/3774224475 >> 172.21.15.22:0/3774224475 conn(0x7f96500fc580 msgr2=0x7f96500fd2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:45:35.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:35.033+0000 7f9657a3d700 1 -- 172.21.15.22:0/3774224475 shutdown_connections 2024-03-24T09:45:35.035 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:35.033+0000 7f9657a3d700 1 -- 172.21.15.22:0/3774224475 wait complete. 2024-03-24T09:45:35.765 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-24T09:45:35.766 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-24T09:45:35.766 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-24T09:45:35.767 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-24T09:45:35.768 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-24T09:45:35.768 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-24T09:45:35.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:35 smithi155 ceph-mon[37016]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:35.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:35 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/3774224475' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:45:36.024 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:45:36.031 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:45:36.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:35 smithi022 ceph-mon[29512]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:36.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:35 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/3774224475' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:45:37.955 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:37 smithi022 ceph-mon[29512]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:37.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:37 smithi155 ceph-mon[37016]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:38.585 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:38 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:45:38.586 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:45:38.592 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:45:38.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:38 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:45:39.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.055+0000 7fab25677700 1 -- 172.21.15.22:0/1192222294 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fab200731e0 msgr2=0x7fab200735c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:39.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.055+0000 7fab25677700 1 --2- 172.21.15.22:0/1192222294 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fab200731e0 0x7fab200735c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fab10009940 tx=0x7fab10009c50 comp rx=0 tx=0).stop 2024-03-24T09:45:39.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.055+0000 7fab25677700 1 -- 172.21.15.22:0/1192222294 shutdown_connections 2024-03-24T09:45:39.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.055+0000 7fab25677700 1 --2- 172.21.15.22:0/1192222294 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab20073b90 0x7fab2010cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:39.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.055+0000 7fab25677700 1 --2- 172.21.15.22:0/1192222294 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fab200731e0 0x7fab200735c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:39.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.055+0000 7fab25677700 1 -- 172.21.15.22:0/1192222294 >> 172.21.15.22:0/1192222294 conn(0x7fab200fc5a0 msgr2=0x7fab200fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:45:39.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.055+0000 7fab25677700 1 -- 172.21.15.22:0/1192222294 shutdown_connections 2024-03-24T09:45:39.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.055+0000 7fab25677700 1 -- 172.21.15.22:0/1192222294 wait complete. 2024-03-24T09:45:39.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.055+0000 7fab25677700 1 Processor -- start 2024-03-24T09:45:39.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.056+0000 7fab25677700 1 -- start start 2024-03-24T09:45:39.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.056+0000 7fab25677700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab200731e0 0x7fab2019e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:39.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.056+0000 7fab25677700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fab20073b90 0x7fab2019ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:39.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.056+0000 7fab25677700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab2019f400 con 0x7fab20073b90 2024-03-24T09:45:39.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.056+0000 7fab25677700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab20198a00 con 0x7fab200731e0 2024-03-24T09:45:39.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.056+0000 7fab1e7fc700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fab20073b90 0x7fab2019ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:39.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.056+0000 7fab1e7fc700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fab20073b90 0x7fab2019ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42172/0 (socket says 172.21.15.22:42172) 2024-03-24T09:45:39.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.056+0000 7fab1e7fc700 1 -- 172.21.15.22:0/2308031650 learned_addr learned my addr 172.21.15.22:0/2308031650 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:45:39.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.056+0000 7fab1effd700 1 --2- 172.21.15.22:0/2308031650 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab200731e0 0x7fab2019e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:39.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.056+0000 7fab1e7fc700 1 -- 172.21.15.22:0/2308031650 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab200731e0 msgr2=0x7fab2019e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:39.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.056+0000 7fab1e7fc700 1 --2- 172.21.15.22:0/2308031650 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab200731e0 0x7fab2019e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:39.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.056+0000 7fab1e7fc700 1 -- 172.21.15.22:0/2308031650 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab100095f0 con 0x7fab20073b90 2024-03-24T09:45:39.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.056+0000 7fab1e7fc700 1 --2- 172.21.15.22:0/2308031650 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fab20073b90 0x7fab2019ee00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fab1400c930 tx=0x7fab1400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:39.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.056+0000 7fab07fff700 1 -- 172.21.15.22:0/2308031650 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab14007ad0 con 0x7fab20073b90 2024-03-24T09:45:39.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.057+0000 7fab07fff700 1 -- 172.21.15.22:0/2308031650 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fab1400ce80 con 0x7fab20073b90 2024-03-24T09:45:39.059 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.057+0000 7fab07fff700 1 -- 172.21.15.22:0/2308031650 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab14018630 con 0x7fab20073b90 2024-03-24T09:45:39.060 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.057+0000 7fab25677700 1 -- 172.21.15.22:0/2308031650 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab20198ce0 con 0x7fab20073b90 2024-03-24T09:45:39.060 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.057+0000 7fab25677700 1 -- 172.21.15.22:0/2308031650 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab20199230 con 0x7fab20073b90 2024-03-24T09:45:39.060 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.057+0000 7fab25677700 1 -- 172.21.15.22:0/2308031650 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fab201100c0 con 0x7fab20073b90 2024-03-24T09:45:39.061 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.058+0000 7fab07fff700 1 -- 172.21.15.22:0/2308031650 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7fab14007c30 con 0x7fab20073b90 2024-03-24T09:45:39.061 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.058+0000 7fab07fff700 1 --2- 172.21.15.22:0/2308031650 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fab0806cce0 0x7fab0806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:39.061 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.059+0000 7fab07fff700 1 -- 172.21.15.22:0/2308031650 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fab1408c210 con 0x7fab20073b90 2024-03-24T09:45:39.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.059+0000 7fab1effd700 1 --2- 172.21.15.22:0/2308031650 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fab0806cce0 0x7fab0806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:39.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.059+0000 7fab07fff700 1 --2- 172.21.15.22:0/2308031650 >> [v2:172.21.15.155:6816/3664800012,v1:172.21.15.155:6817/3664800012] conn(0x7fab080729c0 0x7fab08074de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:39.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.059+0000 7fab07fff700 1 -- 172.21.15.22:0/2308031650 --> [v2:172.21.15.155:6816/3664800012,v1:172.21.15.155:6817/3664800012] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fab08075490 con 0x7fab080729c0 2024-03-24T09:45:39.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.059+0000 7fab07fff700 1 -- 172.21.15.22:0/2308031650 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fab1408c690 con 0x7fab20073b90 2024-03-24T09:45:39.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.059+0000 7fab1effd700 1 --2- 172.21.15.22:0/2308031650 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fab0806cce0 0x7fab0806f1a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fab10006010 tx=0x7fab1000b8f0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:39.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.059+0000 7fab1f7fe700 1 --2- 172.21.15.22:0/2308031650 >> [v2:172.21.15.155:6816/3664800012,v1:172.21.15.155:6817/3664800012] conn(0x7fab080729c0 0x7fab08074de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:39.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.060+0000 7fab1f7fe700 1 --2- 172.21.15.22:0/2308031650 >> [v2:172.21.15.155:6816/3664800012,v1:172.21.15.155:6817/3664800012] conn(0x7fab080729c0 0x7fab08074de0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:39.062 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.060+0000 7fab07fff700 1 -- 172.21.15.22:0/2308031650 <== osd.5 v2:172.21.15.155:6816/3664800012 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fab08075490 con 0x7fab080729c0 2024-03-24T09:45:39.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.076+0000 7fab25677700 1 -- 172.21.15.22:0/2308031650 --> [v2:172.21.15.155:6816/3664800012,v1:172.21.15.155:6817/3664800012] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fab201997a0 con 0x7fab080729c0 2024-03-24T09:45:39.076 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.076+0000 7fab07fff700 1 -- 172.21.15.22:0/2308031650 <== osd.5 v2:172.21.15.155:6816/3664800012 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fab201997a0 con 0x7fab080729c0 2024-03-24T09:45:39.077 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.077+0000 7fab25677700 1 -- 172.21.15.22:0/2308031650 >> [v2:172.21.15.155:6816/3664800012,v1:172.21.15.155:6817/3664800012] conn(0x7fab080729c0 msgr2=0x7fab08074de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:39.077 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.077+0000 7fab25677700 1 --2- 172.21.15.22:0/2308031650 >> [v2:172.21.15.155:6816/3664800012,v1:172.21.15.155:6817/3664800012] conn(0x7fab080729c0 0x7fab08074de0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:39.078 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.077+0000 7fab25677700 1 -- 172.21.15.22:0/2308031650 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fab0806cce0 msgr2=0x7fab0806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:39.078 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.077+0000 7fab25677700 1 --2- 172.21.15.22:0/2308031650 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fab0806cce0 0x7fab0806f1a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fab10006010 tx=0x7fab1000b8f0 comp rx=0 tx=0).stop 2024-03-24T09:45:39.078 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.077+0000 7fab25677700 1 -- 172.21.15.22:0/2308031650 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fab20073b90 msgr2=0x7fab2019ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:39.078 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.077+0000 7fab25677700 1 --2- 172.21.15.22:0/2308031650 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fab20073b90 0x7fab2019ee00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fab1400c930 tx=0x7fab1400cc40 comp rx=0 tx=0).stop 2024-03-24T09:45:39.078 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.077+0000 7fab25677700 1 -- 172.21.15.22:0/2308031650 shutdown_connections 2024-03-24T09:45:39.078 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.077+0000 7fab25677700 1 --2- 172.21.15.22:0/2308031650 >> [v2:172.21.15.155:6816/3664800012,v1:172.21.15.155:6817/3664800012] conn(0x7fab080729c0 0x7fab08074de0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:39.078 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.077+0000 7fab25677700 1 --2- 172.21.15.22:0/2308031650 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fab0806cce0 0x7fab0806f1a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:39.078 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.077+0000 7fab25677700 1 --2- 172.21.15.22:0/2308031650 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab200731e0 0x7fab2019e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:39.078 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.077+0000 7fab25677700 1 --2- 172.21.15.22:0/2308031650 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fab20073b90 0x7fab2019ee00 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:39.078 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.077+0000 7fab25677700 1 -- 172.21.15.22:0/2308031650 >> 172.21.15.22:0/2308031650 conn(0x7fab200fc5a0 msgr2=0x7fab201076d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:45:39.078 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.077+0000 7fab25677700 1 -- 172.21.15.22:0/2308031650 shutdown_connections 2024-03-24T09:45:39.079 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.077+0000 7fab25677700 1 -- 172.21.15.22:0/2308031650 wait complete. 2024-03-24T09:45:39.239 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:45:39.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:39 smithi022 ceph-mon[29512]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:39.898 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.898+0000 7fe1a0dc5700 1 -- 172.21.15.22:0/2162772803 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe19c106a70 msgr2=0x7fe19c106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:39.898 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.898+0000 7fe1a0dc5700 1 --2- 172.21.15.22:0/2162772803 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe19c106a70 0x7fe19c106e50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe18c009910 tx=0x7fe18c009c20 comp rx=0 tx=0).stop 2024-03-24T09:45:39.899 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.898+0000 7fe1a0dc5700 1 -- 172.21.15.22:0/2162772803 shutdown_connections 2024-03-24T09:45:39.899 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.898+0000 7fe1a0dc5700 1 --2- 172.21.15.22:0/2162772803 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe19c100a70 0x7fe19c100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:39.899 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.898+0000 7fe1a0dc5700 1 --2- 172.21.15.22:0/2162772803 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe19c106a70 0x7fe19c106e50 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:39.899 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.898+0000 7fe1a0dc5700 1 -- 172.21.15.22:0/2162772803 >> 172.21.15.22:0/2162772803 conn(0x7fe19c0fc5a0 msgr2=0x7fe19c0fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:45:39.899 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.898+0000 7fe1a0dc5700 1 -- 172.21.15.22:0/2162772803 shutdown_connections 2024-03-24T09:45:39.899 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.898+0000 7fe1a0dc5700 1 -- 172.21.15.22:0/2162772803 wait complete. 2024-03-24T09:45:39.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.899+0000 7fe1a0dc5700 1 Processor -- start 2024-03-24T09:45:39.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.899+0000 7fe1a0dc5700 1 -- start start 2024-03-24T09:45:39.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.899+0000 7fe1a0dc5700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe19c100a70 0x7fe19c0730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:39.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.899+0000 7fe1a0dc5700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe19c106a70 0x7fe19c0735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:39.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.899+0000 7fe1a0dc5700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c073b30 con 0x7fe19c100a70 2024-03-24T09:45:39.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.899+0000 7fe1a0dc5700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c073c70 con 0x7fe19c106a70 2024-03-24T09:45:39.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.899+0000 7fe19a59c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe19c100a70 0x7fe19c0730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:39.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.899+0000 7fe19a59c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe19c100a70 0x7fe19c0730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42190/0 (socket says 172.21.15.22:42190) 2024-03-24T09:45:39.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.899+0000 7fe19a59c700 1 -- 172.21.15.22:0/2339504996 learned_addr learned my addr 172.21.15.22:0/2339504996 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:45:39.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.899+0000 7fe199d9b700 1 --2- 172.21.15.22:0/2339504996 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe19c106a70 0x7fe19c0735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:39.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.899+0000 7fe19a59c700 1 -- 172.21.15.22:0/2339504996 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe19c106a70 msgr2=0x7fe19c0735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:39.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.899+0000 7fe19a59c700 1 --2- 172.21.15.22:0/2339504996 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe19c106a70 0x7fe19c0735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:39.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.899+0000 7fe19a59c700 1 -- 172.21.15.22:0/2339504996 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe18c0095f0 con 0x7fe19c100a70 2024-03-24T09:45:39.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.900+0000 7fe19a59c700 1 --2- 172.21.15.22:0/2339504996 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe19c100a70 0x7fe19c0730b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe18c00bd70 tx=0x7fe18c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:39.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.900+0000 7fe18b7fe700 1 -- 172.21.15.22:0/2339504996 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe18c01d070 con 0x7fe19c100a70 2024-03-24T09:45:39.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.900+0000 7fe18b7fe700 1 -- 172.21.15.22:0/2339504996 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe18c0056e0 con 0x7fe19c100a70 2024-03-24T09:45:39.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.900+0000 7fe18b7fe700 1 -- 172.21.15.22:0/2339504996 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe18c017400 con 0x7fe19c100a70 2024-03-24T09:45:39.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.900+0000 7fe1a0dc5700 1 -- 172.21.15.22:0/2339504996 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe19c073ef0 con 0x7fe19c100a70 2024-03-24T09:45:39.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.900+0000 7fe1a0dc5700 1 -- 172.21.15.22:0/2339504996 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe19c1a2d90 con 0x7fe19c100a70 2024-03-24T09:45:39.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.901+0000 7fe1a0dc5700 1 -- 172.21.15.22:0/2339504996 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe19c10df70 con 0x7fe19c100a70 2024-03-24T09:45:39.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.902+0000 7fe18b7fe700 1 -- 172.21.15.22:0/2339504996 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7fe18c003a60 con 0x7fe19c100a70 2024-03-24T09:45:39.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.902+0000 7fe18b7fe700 1 --2- 172.21.15.22:0/2339504996 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fe18406cd30 0x7fe18406f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:39.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.902+0000 7fe18b7fe700 1 -- 172.21.15.22:0/2339504996 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fe18c08d400 con 0x7fe19c100a70 2024-03-24T09:45:39.904 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.903+0000 7fe199d9b700 1 --2- 172.21.15.22:0/2339504996 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fe18406cd30 0x7fe18406f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:39.905 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.903+0000 7fe18b7fe700 1 --2- 172.21.15.22:0/2339504996 >> [v2:172.21.15.155:6800/2405415706,v1:172.21.15.155:6801/2405415706] conn(0x7fe184072a10 0x7fe184074e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:39.905 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.903+0000 7fe18b7fe700 1 -- 172.21.15.22:0/2339504996 --> [v2:172.21.15.155:6800/2405415706,v1:172.21.15.155:6801/2405415706] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe1840754e0 con 0x7fe184072a10 2024-03-24T09:45:39.905 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.903+0000 7fe18b7fe700 1 -- 172.21.15.22:0/2339504996 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe18c08fa20 con 0x7fe19c100a70 2024-03-24T09:45:39.905 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.903+0000 7fe19ad9d700 1 --2- 172.21.15.22:0/2339504996 >> [v2:172.21.15.155:6800/2405415706,v1:172.21.15.155:6801/2405415706] conn(0x7fe184072a10 0x7fe184074e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:39.905 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.903+0000 7fe199d9b700 1 --2- 172.21.15.22:0/2339504996 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fe18406cd30 0x7fe18406f1f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe190005950 tx=0x7fe1900058e0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:39.905 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.903+0000 7fe19ad9d700 1 --2- 172.21.15.22:0/2339504996 >> [v2:172.21.15.155:6800/2405415706,v1:172.21.15.155:6801/2405415706] conn(0x7fe184072a10 0x7fe184074e30 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:39.905 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.904+0000 7fe18b7fe700 1 -- 172.21.15.22:0/2339504996 <== osd.3 v2:172.21.15.155:6800/2405415706 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fe1840754e0 con 0x7fe184072a10 2024-03-24T09:45:39.921 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.920+0000 7fe1a0dc5700 1 -- 172.21.15.22:0/2339504996 --> [v2:172.21.15.155:6800/2405415706,v1:172.21.15.155:6801/2405415706] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe19c1021b0 con 0x7fe184072a10 2024-03-24T09:45:39.921 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.921+0000 7fe18b7fe700 1 -- 172.21.15.22:0/2339504996 <== osd.3 v2:172.21.15.155:6800/2405415706 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fe19c1021b0 con 0x7fe184072a10 2024-03-24T09:45:39.922 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.921+0000 7fe1a0dc5700 1 -- 172.21.15.22:0/2339504996 >> [v2:172.21.15.155:6800/2405415706,v1:172.21.15.155:6801/2405415706] conn(0x7fe184072a10 msgr2=0x7fe184074e30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:39.922 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.921+0000 7fe1a0dc5700 1 --2- 172.21.15.22:0/2339504996 >> [v2:172.21.15.155:6800/2405415706,v1:172.21.15.155:6801/2405415706] conn(0x7fe184072a10 0x7fe184074e30 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:39.922 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.921+0000 7fe1a0dc5700 1 -- 172.21.15.22:0/2339504996 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fe18406cd30 msgr2=0x7fe18406f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:39.922 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.921+0000 7fe1a0dc5700 1 --2- 172.21.15.22:0/2339504996 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fe18406cd30 0x7fe18406f1f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe190005950 tx=0x7fe1900058e0 comp rx=0 tx=0).stop 2024-03-24T09:45:39.922 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.921+0000 7fe1a0dc5700 1 -- 172.21.15.22:0/2339504996 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe19c100a70 msgr2=0x7fe19c0730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:39.922 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.921+0000 7fe1a0dc5700 1 --2- 172.21.15.22:0/2339504996 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe19c100a70 0x7fe19c0730b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe18c00bd70 tx=0x7fe18c00be50 comp rx=0 tx=0).stop 2024-03-24T09:45:39.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.921+0000 7fe1a0dc5700 1 -- 172.21.15.22:0/2339504996 shutdown_connections 2024-03-24T09:45:39.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.921+0000 7fe1a0dc5700 1 --2- 172.21.15.22:0/2339504996 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fe18406cd30 0x7fe18406f1f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:39.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.921+0000 7fe1a0dc5700 1 --2- 172.21.15.22:0/2339504996 >> [v2:172.21.15.155:6800/2405415706,v1:172.21.15.155:6801/2405415706] conn(0x7fe184072a10 0x7fe184074e30 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:39.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.921+0000 7fe1a0dc5700 1 --2- 172.21.15.22:0/2339504996 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe19c100a70 0x7fe19c0730b0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:39.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.921+0000 7fe1a0dc5700 1 --2- 172.21.15.22:0/2339504996 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe19c106a70 0x7fe19c0735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:39.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.921+0000 7fe1a0dc5700 1 -- 172.21.15.22:0/2339504996 >> 172.21.15.22:0/2339504996 conn(0x7fe19c0fc5a0 msgr2=0x7fe19c0fd220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:45:39.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.921+0000 7fe1a0dc5700 1 -- 172.21.15.22:0/2339504996 shutdown_connections 2024-03-24T09:45:39.923 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:39.921+0000 7fe1a0dc5700 1 -- 172.21.15.22:0/2339504996 wait complete. 2024-03-24T09:45:39.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:39 smithi155 ceph-mon[37016]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:41.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:41 smithi022 ceph-mon[29512]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:41.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:41 smithi155 ceph-mon[37016]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:42.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.001+0000 7fbb24a3b700 1 -- 172.21.15.22:0/4281914373 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbb200ff950 msgr2=0x7fbb200ffdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:42.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.001+0000 7fbb24a3b700 1 --2- 172.21.15.22:0/4281914373 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbb200ff950 0x7fbb200ffdd0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbb08009910 tx=0x7fbb08009c20 comp rx=0 tx=0).stop 2024-03-24T09:45:42.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.001+0000 7fbb24a3b700 1 -- 172.21.15.22:0/4281914373 shutdown_connections 2024-03-24T09:45:42.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.001+0000 7fbb24a3b700 1 --2- 172.21.15.22:0/4281914373 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbb200ff950 0x7fbb200ffdd0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:42.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.001+0000 7fbb24a3b700 1 --2- 172.21.15.22:0/4281914373 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb20105950 0x7fbb20105d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:42.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.001+0000 7fbb24a3b700 1 -- 172.21.15.22:0/4281914373 >> 172.21.15.22:0/4281914373 conn(0x7fbb20076640 msgr2=0x7fbb200fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:45:42.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.001+0000 7fbb24a3b700 1 -- 172.21.15.22:0/4281914373 shutdown_connections 2024-03-24T09:45:42.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.001+0000 7fbb24a3b700 1 -- 172.21.15.22:0/4281914373 wait complete. 2024-03-24T09:45:42.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.002+0000 7fbb24a3b700 1 Processor -- start 2024-03-24T09:45:42.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.002+0000 7fbb24a3b700 1 -- start start 2024-03-24T09:45:42.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.002+0000 7fbb24a3b700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbb200ff950 0x7fbb200fbaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:42.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.002+0000 7fbb24a3b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb20105950 0x7fbb200fc030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:42.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.002+0000 7fbb24a3b700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb200fc570 con 0x7fbb200ff950 2024-03-24T09:45:42.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.002+0000 7fbb24a3b700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb200fc6b0 con 0x7fbb20105950 2024-03-24T09:45:42.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.002+0000 7fbb1e59c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbb200ff950 0x7fbb200fbaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:42.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.002+0000 7fbb1e59c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbb200ff950 0x7fbb200fbaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42204/0 (socket says 172.21.15.22:42204) 2024-03-24T09:45:42.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.002+0000 7fbb1e59c700 1 -- 172.21.15.22:0/2071720571 learned_addr learned my addr 172.21.15.22:0/2071720571 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:45:42.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.002+0000 7fbb17fff700 1 --2- 172.21.15.22:0/2071720571 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb20105950 0x7fbb200fc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:42.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.002+0000 7fbb1e59c700 1 -- 172.21.15.22:0/2071720571 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb20105950 msgr2=0x7fbb200fc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:42.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.002+0000 7fbb1e59c700 1 --2- 172.21.15.22:0/2071720571 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb20105950 0x7fbb200fc030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:42.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.002+0000 7fbb1e59c700 1 -- 172.21.15.22:0/2071720571 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb080095f0 con 0x7fbb200ff950 2024-03-24T09:45:42.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.002+0000 7fbb1e59c700 1 --2- 172.21.15.22:0/2071720571 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbb200ff950 0x7fbb200fbaf0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fbb1000b4b0 tx=0x7fbb1000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:42.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.003+0000 7fbb177fe700 1 -- 172.21.15.22:0/2071720571 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb10011800 con 0x7fbb200ff950 2024-03-24T09:45:42.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.003+0000 7fbb24a3b700 1 -- 172.21.15.22:0/2071720571 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb201a2c70 con 0x7fbb200ff950 2024-03-24T09:45:42.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.003+0000 7fbb177fe700 1 -- 172.21.15.22:0/2071720571 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbb10011e20 con 0x7fbb200ff950 2024-03-24T09:45:42.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.003+0000 7fbb177fe700 1 -- 172.21.15.22:0/2071720571 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb1000f2e0 con 0x7fbb200ff950 2024-03-24T09:45:42.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.003+0000 7fbb24a3b700 1 -- 172.21.15.22:0/2071720571 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb201a2fd0 con 0x7fbb200ff950 2024-03-24T09:45:42.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.003+0000 7fbb24a3b700 1 -- 172.21.15.22:0/2071720571 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbb2010ced0 con 0x7fbb200ff950 2024-03-24T09:45:42.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.004+0000 7fbb177fe700 1 -- 172.21.15.22:0/2071720571 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7fbb10010380 con 0x7fbb200ff950 2024-03-24T09:45:42.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.004+0000 7fbb177fe700 1 --2- 172.21.15.22:0/2071720571 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fbb0c06cce0 0x7fbb0c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:42.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.005+0000 7fbb17fff700 1 --2- 172.21.15.22:0/2071720571 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fbb0c06cce0 0x7fbb0c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:42.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.005+0000 7fbb177fe700 1 -- 172.21.15.22:0/2071720571 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fbb1008bad0 con 0x7fbb200ff950 2024-03-24T09:45:42.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.005+0000 7fbb177fe700 1 --2- 172.21.15.22:0/2071720571 >> [v2:172.21.15.22:6802/3431970421,v1:172.21.15.22:6803/3431970421] conn(0x7fbb0c0729c0 0x7fbb0c074de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:42.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.005+0000 7fbb177fe700 1 -- 172.21.15.22:0/2071720571 --> [v2:172.21.15.22:6802/3431970421,v1:172.21.15.22:6803/3431970421] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb0c075490 con 0x7fbb0c0729c0 2024-03-24T09:45:42.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.005+0000 7fbb177fe700 1 -- 172.21.15.22:0/2071720571 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbb100902a0 con 0x7fbb200ff950 2024-03-24T09:45:42.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.005+0000 7fbb1ed9d700 1 --2- 172.21.15.22:0/2071720571 >> [v2:172.21.15.22:6802/3431970421,v1:172.21.15.22:6803/3431970421] conn(0x7fbb0c0729c0 0x7fbb0c074de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:42.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.005+0000 7fbb17fff700 1 --2- 172.21.15.22:0/2071720571 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fbb0c06cce0 0x7fbb0c06f1a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbb080059d0 tx=0x7fbb0801a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:42.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.005+0000 7fbb1ed9d700 1 --2- 172.21.15.22:0/2071720571 >> [v2:172.21.15.22:6802/3431970421,v1:172.21.15.22:6803/3431970421] conn(0x7fbb0c0729c0 0x7fbb0c074de0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:42.006 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.006+0000 7fbb177fe700 1 -- 172.21.15.22:0/2071720571 <== osd.0 v2:172.21.15.22:6802/3431970421 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fbb0c075490 con 0x7fbb0c0729c0 2024-03-24T09:45:42.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.021+0000 7fbb24a3b700 1 -- 172.21.15.22:0/2071720571 --> [v2:172.21.15.22:6802/3431970421,v1:172.21.15.22:6803/3431970421] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbb20101090 con 0x7fbb0c0729c0 2024-03-24T09:45:42.022 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.022+0000 7fbb177fe700 1 -- 172.21.15.22:0/2071720571 <== osd.0 v2:172.21.15.22:6802/3431970421 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fbb20101090 con 0x7fbb0c0729c0 2024-03-24T09:45:42.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.022+0000 7fbb24a3b700 1 -- 172.21.15.22:0/2071720571 >> [v2:172.21.15.22:6802/3431970421,v1:172.21.15.22:6803/3431970421] conn(0x7fbb0c0729c0 msgr2=0x7fbb0c074de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:42.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.022+0000 7fbb24a3b700 1 --2- 172.21.15.22:0/2071720571 >> [v2:172.21.15.22:6802/3431970421,v1:172.21.15.22:6803/3431970421] conn(0x7fbb0c0729c0 0x7fbb0c074de0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:42.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.022+0000 7fbb24a3b700 1 -- 172.21.15.22:0/2071720571 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fbb0c06cce0 msgr2=0x7fbb0c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:42.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.022+0000 7fbb24a3b700 1 --2- 172.21.15.22:0/2071720571 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fbb0c06cce0 0x7fbb0c06f1a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbb080059d0 tx=0x7fbb0801a040 comp rx=0 tx=0).stop 2024-03-24T09:45:42.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.022+0000 7fbb24a3b700 1 -- 172.21.15.22:0/2071720571 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbb200ff950 msgr2=0x7fbb200fbaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:42.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.022+0000 7fbb24a3b700 1 --2- 172.21.15.22:0/2071720571 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbb200ff950 0x7fbb200fbaf0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fbb1000b4b0 tx=0x7fbb1000b870 comp rx=0 tx=0).stop 2024-03-24T09:45:42.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.022+0000 7fbb24a3b700 1 -- 172.21.15.22:0/2071720571 shutdown_connections 2024-03-24T09:45:42.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.022+0000 7fbb24a3b700 1 --2- 172.21.15.22:0/2071720571 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fbb0c06cce0 0x7fbb0c06f1a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:42.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.022+0000 7fbb24a3b700 1 --2- 172.21.15.22:0/2071720571 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbb200ff950 0x7fbb200fbaf0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:42.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.022+0000 7fbb24a3b700 1 --2- 172.21.15.22:0/2071720571 >> [v2:172.21.15.22:6802/3431970421,v1:172.21.15.22:6803/3431970421] conn(0x7fbb0c0729c0 0x7fbb0c074de0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:42.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.022+0000 7fbb24a3b700 1 --2- 172.21.15.22:0/2071720571 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb20105950 0x7fbb200fc030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:42.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.022+0000 7fbb24a3b700 1 -- 172.21.15.22:0/2071720571 >> 172.21.15.22:0/2071720571 conn(0x7fbb20076640 msgr2=0x7fbb200fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:45:42.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.022+0000 7fbb24a3b700 1 -- 172.21.15.22:0/2071720571 shutdown_connections 2024-03-24T09:45:42.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:42.022+0000 7fbb24a3b700 1 -- 172.21.15.22:0/2071720571 wait complete. 2024-03-24T09:45:42.319 INFO:teuthology.orchestra.run.smithi022.stdout:111669149713 2024-03-24T09:45:42.319 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-24T09:45:42.496 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:45:42.919 INFO:teuthology.orchestra.run.smithi022.stdout:167503724553 2024-03-24T09:45:42.919 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-24T09:45:43.891 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:43 smithi022 ceph-mon[29512]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:43.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:43 smithi155 ceph-mon[37016]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:45.179 INFO:teuthology.orchestra.run.smithi022.stdout:38654705699 2024-03-24T09:45:45.180 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-24T09:45:45.337 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.336+0000 7fb1664f7700 1 -- 172.21.15.22:0/287311786 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb1601059a0 msgr2=0x7fb160105d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:45.337 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.336+0000 7fb1664f7700 1 --2- 172.21.15.22:0/287311786 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb1601059a0 0x7fb160105d80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb150009910 tx=0x7fb150009c20 comp rx=0 tx=0).stop 2024-03-24T09:45:45.337 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.336+0000 7fb1664f7700 1 -- 172.21.15.22:0/287311786 shutdown_connections 2024-03-24T09:45:45.337 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.336+0000 7fb1664f7700 1 --2- 172.21.15.22:0/287311786 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1600ff9a0 0x7fb1600ffe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:45.337 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.336+0000 7fb1664f7700 1 --2- 172.21.15.22:0/287311786 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb1601059a0 0x7fb160105d80 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:45.338 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.336+0000 7fb1664f7700 1 -- 172.21.15.22:0/287311786 >> 172.21.15.22:0/287311786 conn(0x7fb1600fb4b0 msgr2=0x7fb1600fd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:45:45.338 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.336+0000 7fb1664f7700 1 -- 172.21.15.22:0/287311786 shutdown_connections 2024-03-24T09:45:45.338 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.336+0000 7fb1664f7700 1 -- 172.21.15.22:0/287311786 wait complete. 2024-03-24T09:45:45.338 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.336+0000 7fb1664f7700 1 Processor -- start 2024-03-24T09:45:45.338 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.337+0000 7fb1664f7700 1 -- start start 2024-03-24T09:45:45.338 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.337+0000 7fb1664f7700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb1600ff9a0 0x7fb16019c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:45.338 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.337+0000 7fb1664f7700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1601059a0 0x7fb16019cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:45.338 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.337+0000 7fb1664f7700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb16019d2a0 con 0x7fb1600ff9a0 2024-03-24T09:45:45.338 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.337+0000 7fb1664f7700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb160196860 con 0x7fb1601059a0 2024-03-24T09:45:45.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.337+0000 7fb15ffff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb1600ff9a0 0x7fb16019c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:45.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.337+0000 7fb15ffff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb1600ff9a0 0x7fb16019c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42216/0 (socket says 172.21.15.22:42216) 2024-03-24T09:45:45.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.337+0000 7fb15ffff700 1 -- 172.21.15.22:0/1783512097 learned_addr learned my addr 172.21.15.22:0/1783512097 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:45:45.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.337+0000 7fb15f7fe700 1 --2- 172.21.15.22:0/1783512097 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1601059a0 0x7fb16019cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:45.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.337+0000 7fb15ffff700 1 -- 172.21.15.22:0/1783512097 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1601059a0 msgr2=0x7fb16019cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:45.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.337+0000 7fb15ffff700 1 --2- 172.21.15.22:0/1783512097 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1601059a0 0x7fb16019cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:45.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.337+0000 7fb15ffff700 1 -- 172.21.15.22:0/1783512097 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1500095f0 con 0x7fb1600ff9a0 2024-03-24T09:45:45.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.337+0000 7fb15f7fe700 1 --2- 172.21.15.22:0/1783512097 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1601059a0 0x7fb16019cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T09:45:45.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.337+0000 7fb15ffff700 1 --2- 172.21.15.22:0/1783512097 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb1600ff9a0 0x7fb16019c6d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb15000bd70 tx=0x7fb15000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:45.339 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.338+0000 7fb15d7fa700 1 -- 172.21.15.22:0/1783512097 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb15001d070 con 0x7fb1600ff9a0 2024-03-24T09:45:45.340 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.338+0000 7fb15d7fa700 1 -- 172.21.15.22:0/1783512097 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb1500056e0 con 0x7fb1600ff9a0 2024-03-24T09:45:45.340 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.338+0000 7fb1664f7700 1 -- 172.21.15.22:0/1783512097 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb160196ae0 con 0x7fb1600ff9a0 2024-03-24T09:45:45.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.338+0000 7fb15d7fa700 1 -- 172.21.15.22:0/1783512097 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb150017400 con 0x7fb1600ff9a0 2024-03-24T09:45:45.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.338+0000 7fb1664f7700 1 -- 172.21.15.22:0/1783512097 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb160196fd0 con 0x7fb1600ff9a0 2024-03-24T09:45:45.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.339+0000 7fb15d7fa700 1 -- 172.21.15.22:0/1783512097 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7fb150003a60 con 0x7fb1600ff9a0 2024-03-24T09:45:45.341 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.339+0000 7fb1664f7700 1 -- 172.21.15.22:0/1783512097 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb16010cea0 con 0x7fb1600ff9a0 2024-03-24T09:45:45.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.340+0000 7fb15d7fa700 1 --2- 172.21.15.22:0/1783512097 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fb14806cae0 0x7fb14806efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:45.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.340+0000 7fb15f7fe700 1 --2- 172.21.15.22:0/1783512097 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fb14806cae0 0x7fb14806efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:45.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.340+0000 7fb15d7fa700 1 -- 172.21.15.22:0/1783512097 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fb15008edd0 con 0x7fb1600ff9a0 2024-03-24T09:45:45.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.340+0000 7fb15d7fa700 1 --2- 172.21.15.22:0/1783512097 >> [v2:172.21.15.22:6810/1675089147,v1:172.21.15.22:6811/1675089147] conn(0x7fb148072650 0x7fb148074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:45.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.340+0000 7fb15d7fa700 1 -- 172.21.15.22:0/1783512097 --> [v2:172.21.15.22:6810/1675089147,v1:172.21.15.22:6811/1675089147] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb148075120 con 0x7fb148072650 2024-03-24T09:45:45.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.340+0000 7fb15d7fa700 1 -- 172.21.15.22:0/1783512097 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb150096520 con 0x7fb1600ff9a0 2024-03-24T09:45:45.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.340+0000 7fb164a94700 1 --2- 172.21.15.22:0/1783512097 >> [v2:172.21.15.22:6810/1675089147,v1:172.21.15.22:6811/1675089147] conn(0x7fb148072650 0x7fb148074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:45.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.341+0000 7fb15f7fe700 1 --2- 172.21.15.22:0/1783512097 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fb14806cae0 0x7fb14806efa0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb154005fd0 tx=0x7fb154005ee0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:45.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.341+0000 7fb164a94700 1 --2- 172.21.15.22:0/1783512097 >> [v2:172.21.15.22:6810/1675089147,v1:172.21.15.22:6811/1675089147] conn(0x7fb148072650 0x7fb148074a70 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:45.342 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.342+0000 7fb15d7fa700 1 -- 172.21.15.22:0/1783512097 <== osd.1 v2:172.21.15.22:6810/1675089147 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fb148075120 con 0x7fb148072650 2024-03-24T09:45:45.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.357+0000 7fb1664f7700 1 -- 172.21.15.22:0/1783512097 --> [v2:172.21.15.22:6810/1675089147,v1:172.21.15.22:6811/1675089147] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb1601974d0 con 0x7fb148072650 2024-03-24T09:45:45.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.358+0000 7fb15d7fa700 1 -- 172.21.15.22:0/1783512097 <== osd.1 v2:172.21.15.22:6810/1675089147 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb1601974d0 con 0x7fb148072650 2024-03-24T09:45:45.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.358+0000 7fb1664f7700 1 -- 172.21.15.22:0/1783512097 >> [v2:172.21.15.22:6810/1675089147,v1:172.21.15.22:6811/1675089147] conn(0x7fb148072650 msgr2=0x7fb148074a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:45.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.358+0000 7fb1664f7700 1 --2- 172.21.15.22:0/1783512097 >> [v2:172.21.15.22:6810/1675089147,v1:172.21.15.22:6811/1675089147] conn(0x7fb148072650 0x7fb148074a70 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:45.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.358+0000 7fb1664f7700 1 -- 172.21.15.22:0/1783512097 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fb14806cae0 msgr2=0x7fb14806efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:45.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.358+0000 7fb1664f7700 1 --2- 172.21.15.22:0/1783512097 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fb14806cae0 0x7fb14806efa0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb154005fd0 tx=0x7fb154005ee0 comp rx=0 tx=0).stop 2024-03-24T09:45:45.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.358+0000 7fb1664f7700 1 -- 172.21.15.22:0/1783512097 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb1600ff9a0 msgr2=0x7fb16019c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:45.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.358+0000 7fb1664f7700 1 --2- 172.21.15.22:0/1783512097 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb1600ff9a0 0x7fb16019c6d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb15000bd70 tx=0x7fb15000be50 comp rx=0 tx=0).stop 2024-03-24T09:45:45.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.358+0000 7fb1664f7700 1 -- 172.21.15.22:0/1783512097 shutdown_connections 2024-03-24T09:45:45.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.358+0000 7fb1664f7700 1 --2- 172.21.15.22:0/1783512097 >> [v2:172.21.15.22:6810/1675089147,v1:172.21.15.22:6811/1675089147] conn(0x7fb148072650 0x7fb148074a70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:45.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.358+0000 7fb1664f7700 1 --2- 172.21.15.22:0/1783512097 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fb14806cae0 0x7fb14806efa0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:45.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.358+0000 7fb1664f7700 1 --2- 172.21.15.22:0/1783512097 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb1600ff9a0 0x7fb16019c6d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:45.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.358+0000 7fb1664f7700 1 --2- 172.21.15.22:0/1783512097 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1601059a0 0x7fb16019cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:45.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.358+0000 7fb1664f7700 1 -- 172.21.15.22:0/1783512097 >> 172.21.15.22:0/1783512097 conn(0x7fb1600fb4b0 msgr2=0x7fb160104cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:45:45.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.359+0000 7fb1664f7700 1 -- 172.21.15.22:0/1783512097 shutdown_connections 2024-03-24T09:45:45.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:45.359+0000 7fb1664f7700 1 -- 172.21.15.22:0/1783512097 wait complete. 2024-03-24T09:45:45.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:45 smithi022 ceph-mon[29512]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:45.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:45 smithi155 ceph-mon[37016]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:46.881 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:45:46.910 INFO:teuthology.orchestra.run.smithi022.stdout:60129542175 2024-03-24T09:45:46.910 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-24T09:45:47.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.175+0000 7f5752805700 1 -- 172.21.15.22:0/2003563053 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f574c100f60 msgr2=0x7f574c1045e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:47.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.175+0000 7f5752805700 1 --2- 172.21.15.22:0/2003563053 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f574c100f60 0x7f574c1045e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5740009910 tx=0x7f5740009c20 comp rx=0 tx=0).stop 2024-03-24T09:45:47.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.175+0000 7f5752805700 1 -- 172.21.15.22:0/2003563053 shutdown_connections 2024-03-24T09:45:47.175 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.175+0000 7f5752805700 1 --2- 172.21.15.22:0/2003563053 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f574c100f60 0x7f574c1045e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:47.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.175+0000 7f5752805700 1 --2- 172.21.15.22:0/2003563053 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f574c1005b0 0x7f574c100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:47.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.175+0000 7f5752805700 1 -- 172.21.15.22:0/2003563053 >> 172.21.15.22:0/2003563053 conn(0x7f574c074cc0 msgr2=0x7f574c0750d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:45:47.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.175+0000 7f5752805700 1 -- 172.21.15.22:0/2003563053 shutdown_connections 2024-03-24T09:45:47.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.175+0000 7f5752805700 1 -- 172.21.15.22:0/2003563053 wait complete. 2024-03-24T09:45:47.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.176+0000 7f5752805700 1 Processor -- start 2024-03-24T09:45:47.176 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.176+0000 7f5752805700 1 -- start start 2024-03-24T09:45:47.181 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.176+0000 7f5752805700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f574c1005b0 0x7f574c101680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:47.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.176+0000 7f5752805700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f574c100f60 0x7f574c101bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:47.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.176+0000 7f5752805700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574c103650 con 0x7f574c1005b0 2024-03-24T09:45:47.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.176+0000 7f5752805700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574c102100 con 0x7f574c100f60 2024-03-24T09:45:47.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.176+0000 7f574bfff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f574c1005b0 0x7f574c101680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:47.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.176+0000 7f574bfff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f574c1005b0 0x7f574c101680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42236/0 (socket says 172.21.15.22:42236) 2024-03-24T09:45:47.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.176+0000 7f574bfff700 1 -- 172.21.15.22:0/4007311007 learned_addr learned my addr 172.21.15.22:0/4007311007 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:45:47.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.177+0000 7f574b7fe700 1 --2- 172.21.15.22:0/4007311007 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f574c100f60 0x7f574c101bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:47.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.177+0000 7f574bfff700 1 -- 172.21.15.22:0/4007311007 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f574c100f60 msgr2=0x7f574c101bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:47.182 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.177+0000 7f574bfff700 1 --2- 172.21.15.22:0/4007311007 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f574c100f60 0x7f574c101bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:47.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.177+0000 7f574bfff700 1 -- 172.21.15.22:0/4007311007 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57400095f0 con 0x7f574c1005b0 2024-03-24T09:45:47.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.177+0000 7f574bfff700 1 --2- 172.21.15.22:0/4007311007 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f574c1005b0 0x7f574c101680 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f573c00c930 tx=0x7f573c00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:47.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.177+0000 7f57497fa700 1 -- 172.21.15.22:0/4007311007 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f573c007ad0 con 0x7f574c1005b0 2024-03-24T09:45:47.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.177+0000 7f57497fa700 1 -- 172.21.15.22:0/4007311007 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f573c007c30 con 0x7f574c1005b0 2024-03-24T09:45:47.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.177+0000 7f57497fa700 1 -- 172.21.15.22:0/4007311007 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f573c018610 con 0x7f574c1005b0 2024-03-24T09:45:47.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.177+0000 7f5752805700 1 -- 172.21.15.22:0/4007311007 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f574c1023b0 con 0x7f574c1005b0 2024-03-24T09:45:47.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.177+0000 7f5752805700 1 -- 172.21.15.22:0/4007311007 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f574c1a2d80 con 0x7f574c1005b0 2024-03-24T09:45:47.183 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.178+0000 7f5752805700 1 -- 172.21.15.22:0/4007311007 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f574c10ce80 con 0x7f574c1005b0 2024-03-24T09:45:47.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.179+0000 7f57497fa700 1 -- 172.21.15.22:0/4007311007 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f573c00f450 con 0x7f574c1005b0 2024-03-24T09:45:47.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.179+0000 7f57497fa700 1 --2- 172.21.15.22:0/4007311007 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f573406cce0 0x7f573406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:47.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.179+0000 7f57497fa700 1 -- 172.21.15.22:0/4007311007 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f573c08c380 con 0x7f574c1005b0 2024-03-24T09:45:47.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.179+0000 7f574b7fe700 1 --2- 172.21.15.22:0/4007311007 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f573406cce0 0x7f573406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:47.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.179+0000 7f57497fa700 1 --2- 172.21.15.22:0/4007311007 >> [v2:172.21.15.22:6818/1538987315,v1:172.21.15.22:6819/1538987315] conn(0x7f57340729c0 0x7f5734074de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:47.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.180+0000 7f57497fa700 1 -- 172.21.15.22:0/4007311007 --> [v2:172.21.15.22:6818/1538987315,v1:172.21.15.22:6819/1538987315] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5734075490 con 0x7f57340729c0 2024-03-24T09:45:47.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.180+0000 7f57497fa700 1 -- 172.21.15.22:0/4007311007 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f573c08c800 con 0x7f574c1005b0 2024-03-24T09:45:47.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.180+0000 7f5750da2700 1 --2- 172.21.15.22:0/4007311007 >> [v2:172.21.15.22:6818/1538987315,v1:172.21.15.22:6819/1538987315] conn(0x7f57340729c0 0x7f5734074de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:47.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.180+0000 7f574b7fe700 1 --2- 172.21.15.22:0/4007311007 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f573406cce0 0x7f573406f1a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f574c102db0 tx=0x7f574001a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:47.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.180+0000 7f5750da2700 1 --2- 172.21.15.22:0/4007311007 >> [v2:172.21.15.22:6818/1538987315,v1:172.21.15.22:6819/1538987315] conn(0x7f57340729c0 0x7f5734074de0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:47.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.181+0000 7f57497fa700 1 -- 172.21.15.22:0/4007311007 <== osd.2 v2:172.21.15.22:6818/1538987315 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f5734075490 con 0x7f57340729c0 2024-03-24T09:45:47.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.197+0000 7f5752805700 1 -- 172.21.15.22:0/4007311007 --> [v2:172.21.15.22:6818/1538987315,v1:172.21.15.22:6819/1538987315] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f574c068db0 con 0x7f57340729c0 2024-03-24T09:45:47.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.198+0000 7f57497fa700 1 -- 172.21.15.22:0/4007311007 <== osd.2 v2:172.21.15.22:6818/1538987315 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f574c068db0 con 0x7f57340729c0 2024-03-24T09:45:47.198 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.198+0000 7f5752805700 1 -- 172.21.15.22:0/4007311007 >> [v2:172.21.15.22:6818/1538987315,v1:172.21.15.22:6819/1538987315] conn(0x7f57340729c0 msgr2=0x7f5734074de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:47.198 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.198+0000 7f5752805700 1 --2- 172.21.15.22:0/4007311007 >> [v2:172.21.15.22:6818/1538987315,v1:172.21.15.22:6819/1538987315] conn(0x7f57340729c0 0x7f5734074de0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:47.198 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.198+0000 7f5752805700 1 -- 172.21.15.22:0/4007311007 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f573406cce0 msgr2=0x7f573406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:47.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.198+0000 7f5752805700 1 --2- 172.21.15.22:0/4007311007 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f573406cce0 0x7f573406f1a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f574c102db0 tx=0x7f574001a040 comp rx=0 tx=0).stop 2024-03-24T09:45:47.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.198+0000 7f5752805700 1 -- 172.21.15.22:0/4007311007 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f574c1005b0 msgr2=0x7f574c101680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:47.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.198+0000 7f5752805700 1 --2- 172.21.15.22:0/4007311007 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f574c1005b0 0x7f574c101680 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f573c00c930 tx=0x7f573c00ccf0 comp rx=0 tx=0).stop 2024-03-24T09:45:47.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.198+0000 7f5752805700 1 -- 172.21.15.22:0/4007311007 shutdown_connections 2024-03-24T09:45:47.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.198+0000 7f5752805700 1 --2- 172.21.15.22:0/4007311007 >> [v2:172.21.15.22:6818/1538987315,v1:172.21.15.22:6819/1538987315] conn(0x7f57340729c0 0x7f5734074de0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:47.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.198+0000 7f5752805700 1 --2- 172.21.15.22:0/4007311007 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f573406cce0 0x7f573406f1a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:47.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.198+0000 7f5752805700 1 --2- 172.21.15.22:0/4007311007 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f574c1005b0 0x7f574c101680 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:47.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.198+0000 7f5752805700 1 --2- 172.21.15.22:0/4007311007 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f574c100f60 0x7f574c101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:47.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.198+0000 7f5752805700 1 -- 172.21.15.22:0/4007311007 >> 172.21.15.22:0/4007311007 conn(0x7f574c074cc0 msgr2=0x7f574c103ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:45:47.199 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.198+0000 7f5752805700 1 -- 172.21.15.22:0/4007311007 shutdown_connections 2024-03-24T09:45:47.200 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:47.198+0000 7f5752805700 1 -- 172.21.15.22:0/4007311007 wait complete. 2024-03-24T09:45:47.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:47 smithi022 ceph-mon[29512]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:47.843 INFO:teuthology.orchestra.run.smithi022.stdout:81604378649 2024-03-24T09:45:47.843 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-24T09:45:47.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:47 smithi155 ceph-mon[37016]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:49.026 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:45:49.056 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:45:49.058 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:45:49.802 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:49 smithi022 ceph-mon[29512]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:49.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:49 smithi155 ceph-mon[37016]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:51.142 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.143+0000 7f391dae0700 1 -- 172.21.15.22:0/2104477696 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3918101c10 msgr2=0x7f3918101ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:51.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.143+0000 7f391dae0700 1 --2- 172.21.15.22:0/2104477696 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3918101c10 0x7f3918101ff0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3908009910 tx=0x7f3908009c20 comp rx=0 tx=0).stop 2024-03-24T09:45:51.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.143+0000 7f391dae0700 1 -- 172.21.15.22:0/2104477696 shutdown_connections 2024-03-24T09:45:51.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.143+0000 7f391dae0700 1 --2- 172.21.15.22:0/2104477696 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3918102530 0x7f391810aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:51.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.143+0000 7f391dae0700 1 --2- 172.21.15.22:0/2104477696 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3918101c10 0x7f3918101ff0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:51.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.143+0000 7f391dae0700 1 -- 172.21.15.22:0/2104477696 >> 172.21.15.22:0/2104477696 conn(0x7f3918076640 msgr2=0x7f39180fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:45:51.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.143+0000 7f391dae0700 1 -- 172.21.15.22:0/2104477696 shutdown_connections 2024-03-24T09:45:51.143 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.143+0000 7f391dae0700 1 -- 172.21.15.22:0/2104477696 wait complete. 2024-03-24T09:45:51.144 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.144+0000 7f391dae0700 1 Processor -- start 2024-03-24T09:45:51.144 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.144+0000 7f391dae0700 1 -- start start 2024-03-24T09:45:51.145 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.144+0000 7f391dae0700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3918101c10 0x7f3918196a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:51.145 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.144+0000 7f391dae0700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3918102530 0x7f3918196f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:51.145 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.144+0000 7f391dae0700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3918197610 con 0x7f3918102530 2024-03-24T09:45:51.145 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.144+0000 7f391dae0700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f391819b360 con 0x7f3918101c10 2024-03-24T09:45:51.145 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.144+0000 7f390ffff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3918102530 0x7f3918196f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:51.145 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.144+0000 7f390ffff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3918102530 0x7f3918196f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:35674/0 (socket says 172.21.15.22:35674) 2024-03-24T09:45:51.145 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.144+0000 7f390ffff700 1 -- 172.21.15.22:0/554047160 learned_addr learned my addr 172.21.15.22:0/554047160 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:45:51.145 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.144+0000 7f39177fe700 1 --2- 172.21.15.22:0/554047160 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3918101c10 0x7f3918196a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:51.145 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.145+0000 7f390ffff700 1 -- 172.21.15.22:0/554047160 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3918101c10 msgr2=0x7f3918196a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:51.145 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.145+0000 7f390ffff700 1 --2- 172.21.15.22:0/554047160 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3918101c10 0x7f3918196a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:51.145 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.145+0000 7f390ffff700 1 -- 172.21.15.22:0/554047160 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39080095f0 con 0x7f3918102530 2024-03-24T09:45:51.145 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.145+0000 7f39177fe700 1 --2- 172.21.15.22:0/554047160 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3918101c10 0x7f3918196a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:45:51.145 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.145+0000 7f390ffff700 1 --2- 172.21.15.22:0/554047160 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3918102530 0x7f3918196f80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f390000d6d0 tx=0x7f390000da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:51.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.145+0000 7f39157fa700 1 -- 172.21.15.22:0/554047160 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f390000f860 con 0x7f3918102530 2024-03-24T09:45:51.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.145+0000 7f39157fa700 1 -- 172.21.15.22:0/554047160 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f390000fe80 con 0x7f3918102530 2024-03-24T09:45:51.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.145+0000 7f391dae0700 1 -- 172.21.15.22:0/554047160 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f391819b640 con 0x7f3918102530 2024-03-24T09:45:51.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.145+0000 7f39157fa700 1 -- 172.21.15.22:0/554047160 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3900010ea0 con 0x7f3918102530 2024-03-24T09:45:51.146 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.145+0000 7f391dae0700 1 -- 172.21.15.22:0/554047160 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f391819bb90 con 0x7f3918102530 2024-03-24T09:45:51.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.146+0000 7f391dae0700 1 -- 172.21.15.22:0/554047160 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f391810df10 con 0x7f3918102530 2024-03-24T09:45:51.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.147+0000 7f39157fa700 1 -- 172.21.15.22:0/554047160 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f3900010460 con 0x7f3918102530 2024-03-24T09:45:51.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.147+0000 7f39157fa700 1 --2- 172.21.15.22:0/554047160 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f390406ccf0 0x7f390406f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:51.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.147+0000 7f39157fa700 1 -- 172.21.15.22:0/554047160 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f390008bad0 con 0x7f3918102530 2024-03-24T09:45:51.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.147+0000 7f39177fe700 1 --2- 172.21.15.22:0/554047160 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f390406ccf0 0x7f390406f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:51.147 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.147+0000 7f39157fa700 1 --2- 172.21.15.22:0/554047160 >> [v2:172.21.15.155:6808/2965421758,v1:172.21.15.155:6809/2965421758] conn(0x7f39040729d0 0x7f3904074df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:51.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.147+0000 7f39157fa700 1 -- 172.21.15.22:0/554047160 --> [v2:172.21.15.155:6808/2965421758,v1:172.21.15.155:6809/2965421758] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f39040754a0 con 0x7f39040729d0 2024-03-24T09:45:51.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.147+0000 7f39157fa700 1 -- 172.21.15.22:0/554047160 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f39000902a0 con 0x7f3918102530 2024-03-24T09:45:51.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.148+0000 7f3917fff700 1 --2- 172.21.15.22:0/554047160 >> [v2:172.21.15.155:6808/2965421758,v1:172.21.15.155:6809/2965421758] conn(0x7f39040729d0 0x7f3904074df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:51.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.148+0000 7f39177fe700 1 --2- 172.21.15.22:0/554047160 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f390406ccf0 0x7f390406f1b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3908005b40 tx=0x7f390801a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:51.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.148+0000 7f3917fff700 1 --2- 172.21.15.22:0/554047160 >> [v2:172.21.15.155:6808/2965421758,v1:172.21.15.155:6809/2965421758] conn(0x7f39040729d0 0x7f3904074df0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:51.148 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.149+0000 7f39157fa700 1 -- 172.21.15.22:0/554047160 <== osd.4 v2:172.21.15.155:6808/2965421758 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f39040754a0 con 0x7f39040729d0 2024-03-24T09:45:51.165 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.166+0000 7f391dae0700 1 -- 172.21.15.22:0/554047160 --> [v2:172.21.15.155:6808/2965421758,v1:172.21.15.155:6809/2965421758] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f391819bff0 con 0x7f39040729d0 2024-03-24T09:45:51.165 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.166+0000 7f39157fa700 1 -- 172.21.15.22:0/554047160 <== osd.4 v2:172.21.15.155:6808/2965421758 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f391819bff0 con 0x7f39040729d0 2024-03-24T09:45:51.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.167+0000 7f391dae0700 1 -- 172.21.15.22:0/554047160 >> [v2:172.21.15.155:6808/2965421758,v1:172.21.15.155:6809/2965421758] conn(0x7f39040729d0 msgr2=0x7f3904074df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:51.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.167+0000 7f391dae0700 1 --2- 172.21.15.22:0/554047160 >> [v2:172.21.15.155:6808/2965421758,v1:172.21.15.155:6809/2965421758] conn(0x7f39040729d0 0x7f3904074df0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:51.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.167+0000 7f391dae0700 1 -- 172.21.15.22:0/554047160 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f390406ccf0 msgr2=0x7f390406f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:51.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.167+0000 7f391dae0700 1 --2- 172.21.15.22:0/554047160 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f390406ccf0 0x7f390406f1b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3908005b40 tx=0x7f390801a040 comp rx=0 tx=0).stop 2024-03-24T09:45:51.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.167+0000 7f391dae0700 1 -- 172.21.15.22:0/554047160 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3918102530 msgr2=0x7f3918196f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:51.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.167+0000 7f391dae0700 1 --2- 172.21.15.22:0/554047160 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3918102530 0x7f3918196f80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f390000d6d0 tx=0x7f390000da90 comp rx=0 tx=0).stop 2024-03-24T09:45:51.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.167+0000 7f391dae0700 1 -- 172.21.15.22:0/554047160 shutdown_connections 2024-03-24T09:45:51.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.167+0000 7f391dae0700 1 --2- 172.21.15.22:0/554047160 >> [v2:172.21.15.155:6808/2965421758,v1:172.21.15.155:6809/2965421758] conn(0x7f39040729d0 0x7f3904074df0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:51.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.167+0000 7f391dae0700 1 --2- 172.21.15.22:0/554047160 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f390406ccf0 0x7f390406f1b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:51.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.167+0000 7f391dae0700 1 --2- 172.21.15.22:0/554047160 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3918101c10 0x7f3918196a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:51.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.167+0000 7f391dae0700 1 --2- 172.21.15.22:0/554047160 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3918102530 0x7f3918196f80 secure :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f390000d6d0 tx=0x7f390000da90 comp rx=0 tx=0).stop 2024-03-24T09:45:51.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.167+0000 7f391dae0700 1 -- 172.21.15.22:0/554047160 >> 172.21.15.22:0/554047160 conn(0x7f3918076640 msgr2=0x7f3918102f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:45:51.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.167+0000 7f391dae0700 1 -- 172.21.15.22:0/554047160 shutdown_connections 2024-03-24T09:45:51.168 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:51.167+0000 7f391dae0700 1 -- 172.21.15.22:0/554047160 wait complete. 2024-03-24T09:45:51.281 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:51 smithi022 ceph-mon[29512]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:51.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:51 smithi155 ceph-mon[37016]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:52.627 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.625+0000 7fc145597700 1 -- 172.21.15.22:0/1888488951 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc140101c90 msgr2=0x7fc140102070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:52.627 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.625+0000 7fc145597700 1 --2- 172.21.15.22:0/1888488951 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc140101c90 0x7fc140102070 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc13000b0d0 tx=0x7fc13000b3e0 comp rx=0 tx=0).stop 2024-03-24T09:45:52.628 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.625+0000 7fc145597700 1 -- 172.21.15.22:0/1888488951 shutdown_connections 2024-03-24T09:45:52.628 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.625+0000 7fc145597700 1 --2- 172.21.15.22:0/1888488951 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1401025b0 0x7fc14010aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:52.628 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.625+0000 7fc145597700 1 --2- 172.21.15.22:0/1888488951 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc140101c90 0x7fc140102070 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:52.628 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.625+0000 7fc145597700 1 -- 172.21.15.22:0/1888488951 >> 172.21.15.22:0/1888488951 conn(0x7fc1400fb480 msgr2=0x7fc1400fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:45:52.628 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.625+0000 7fc145597700 1 -- 172.21.15.22:0/1888488951 shutdown_connections 2024-03-24T09:45:52.628 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.625+0000 7fc145597700 1 -- 172.21.15.22:0/1888488951 wait complete. 2024-03-24T09:45:52.628 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.626+0000 7fc145597700 1 Processor -- start 2024-03-24T09:45:52.628 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.626+0000 7fc145597700 1 -- start start 2024-03-24T09:45:52.628 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.626+0000 7fc145597700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc140101c90 0x7fc14019c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:52.629 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.626+0000 7fc145597700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc1401025b0 0x7fc14019cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:52.629 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.626+0000 7fc145597700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14019d350 con 0x7fc1401025b0 2024-03-24T09:45:52.629 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.626+0000 7fc145597700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc140196930 con 0x7fc140101c90 2024-03-24T09:45:52.629 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.626+0000 7fc13e7fc700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc1401025b0 0x7fc14019cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:52.629 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.626+0000 7fc13e7fc700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc1401025b0 0x7fc14019cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:35698/0 (socket says 172.21.15.22:35698) 2024-03-24T09:45:52.629 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.626+0000 7fc13e7fc700 1 -- 172.21.15.22:0/2132189414 learned_addr learned my addr 172.21.15.22:0/2132189414 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:45:52.629 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.626+0000 7fc13effd700 1 --2- 172.21.15.22:0/2132189414 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc140101c90 0x7fc14019c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:52.629 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.627+0000 7fc13e7fc700 1 -- 172.21.15.22:0/2132189414 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc140101c90 msgr2=0x7fc14019c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:52.629 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.627+0000 7fc13e7fc700 1 --2- 172.21.15.22:0/2132189414 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc140101c90 0x7fc14019c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:52.629 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.627+0000 7fc13e7fc700 1 -- 172.21.15.22:0/2132189414 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc130009d00 con 0x7fc1401025b0 2024-03-24T09:45:52.630 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.627+0000 7fc13e7fc700 1 --2- 172.21.15.22:0/2132189414 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc1401025b0 0x7fc14019cd30 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc12800b4d0 tx=0x7fc12800b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:52.630 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.627+0000 7fc137fff700 1 -- 172.21.15.22:0/2132189414 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc128011800 con 0x7fc1401025b0 2024-03-24T09:45:52.630 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.627+0000 7fc145597700 1 -- 172.21.15.22:0/2132189414 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc140196c10 con 0x7fc1401025b0 2024-03-24T09:45:52.630 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.627+0000 7fc137fff700 1 -- 172.21.15.22:0/2132189414 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc128011e20 con 0x7fc1401025b0 2024-03-24T09:45:52.630 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.627+0000 7fc137fff700 1 -- 172.21.15.22:0/2132189414 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc128010e70 con 0x7fc1401025b0 2024-03-24T09:45:52.630 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.627+0000 7fc145597700 1 -- 172.21.15.22:0/2132189414 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc140197160 con 0x7fc1401025b0 2024-03-24T09:45:52.630 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.628+0000 7fc145597700 1 -- 172.21.15.22:0/2132189414 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc140105420 con 0x7fc1401025b0 2024-03-24T09:45:52.631 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.628+0000 7fc137fff700 1 -- 172.21.15.22:0/2132189414 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7fc128010380 con 0x7fc1401025b0 2024-03-24T09:45:52.631 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.629+0000 7fc137fff700 1 --2- 172.21.15.22:0/2132189414 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fc12c06cce0 0x7fc12c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:52.632 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.629+0000 7fc137fff700 1 -- 172.21.15.22:0/2132189414 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fc12808b980 con 0x7fc1401025b0 2024-03-24T09:45:52.632 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.629+0000 7fc13effd700 1 --2- 172.21.15.22:0/2132189414 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fc12c06cce0 0x7fc12c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:52.632 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.629+0000 7fc13effd700 1 --2- 172.21.15.22:0/2132189414 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fc12c06cce0 0x7fc12c06f1a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc13000b0a0 tx=0x7fc1300060c0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:52.633 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.634+0000 7fc137fff700 1 -- 172.21.15.22:0/2132189414 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc128059b20 con 0x7fc1401025b0 2024-03-24T09:45:52.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.839+0000 7fc145597700 1 -- 172.21.15.22:0/2132189414 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fc140197df0 con 0x7fc1401025b0 2024-03-24T09:45:52.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.839+0000 7fc137fff700 1 -- 172.21.15.22:0/2132189414 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fc1280596b0 con 0x7fc1401025b0 2024-03-24T09:45:52.839 INFO:teuthology.orchestra.run.smithi022.stdout:167503724555 2024-03-24T09:45:52.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.841+0000 7fc145597700 1 -- 172.21.15.22:0/2132189414 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fc12c06cce0 msgr2=0x7fc12c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:52.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.841+0000 7fc145597700 1 --2- 172.21.15.22:0/2132189414 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fc12c06cce0 0x7fc12c06f1a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc13000b0a0 tx=0x7fc1300060c0 comp rx=0 tx=0).stop 2024-03-24T09:45:52.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.841+0000 7fc145597700 1 -- 172.21.15.22:0/2132189414 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc1401025b0 msgr2=0x7fc14019cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:52.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.841+0000 7fc145597700 1 --2- 172.21.15.22:0/2132189414 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc1401025b0 0x7fc14019cd30 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc12800b4d0 tx=0x7fc12800b7e0 comp rx=0 tx=0).stop 2024-03-24T09:45:52.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.841+0000 7fc145597700 1 -- 172.21.15.22:0/2132189414 shutdown_connections 2024-03-24T09:45:52.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.841+0000 7fc145597700 1 --2- 172.21.15.22:0/2132189414 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fc12c06cce0 0x7fc12c06f1a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:52.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.841+0000 7fc145597700 1 --2- 172.21.15.22:0/2132189414 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc140101c90 0x7fc14019c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:52.841 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.841+0000 7fc145597700 1 --2- 172.21.15.22:0/2132189414 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc1401025b0 0x7fc14019cd30 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:52.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.841+0000 7fc145597700 1 -- 172.21.15.22:0/2132189414 >> 172.21.15.22:0/2132189414 conn(0x7fc1400fb480 msgr2=0x7fc140102f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:45:52.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.841+0000 7fc145597700 1 -- 172.21.15.22:0/2132189414 shutdown_connections 2024-03-24T09:45:52.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:52.841+0000 7fc145597700 1 -- 172.21.15.22:0/2132189414 wait complete. 2024-03-24T09:45:53.719 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:53 smithi022 ceph-mon[29512]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:53.719 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:53 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/2132189414' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T09:45:53.720 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:53 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:45:53.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:53 smithi155 ceph-mon[37016]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:53.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:53 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/2132189414' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T09:45:53.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:53 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:45:54.073 INFO:teuthology.orchestra.run.smithi022.stdout:146028888080 2024-03-24T09:45:54.073 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-24T09:45:54.506 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:45:54.934 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724553 got 167503724555 for osd.5 2024-03-24T09:45:54.935 DEBUG:teuthology.parallel:result is None 2024-03-24T09:45:55.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.127+0000 7f0542f94700 1 -- 172.21.15.22:0/1948968120 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f053c101c90 msgr2=0x7f053c102070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:55.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.127+0000 7f0542f94700 1 --2- 172.21.15.22:0/1948968120 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f053c101c90 0x7f053c102070 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0530009940 tx=0x7f0530009c50 comp rx=0 tx=0).stop 2024-03-24T09:45:55.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.128+0000 7f0542f94700 1 -- 172.21.15.22:0/1948968120 shutdown_connections 2024-03-24T09:45:55.129 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.128+0000 7f0542f94700 1 --2- 172.21.15.22:0/1948968120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f053c1025b0 0x7f053c10aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:55.130 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.128+0000 7f0542f94700 1 --2- 172.21.15.22:0/1948968120 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f053c101c90 0x7f053c102070 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:55.130 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.128+0000 7f0542f94700 1 -- 172.21.15.22:0/1948968120 >> 172.21.15.22:0/1948968120 conn(0x7f053c0fb480 msgr2=0x7f053c0fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:45:55.130 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.128+0000 7f0542f94700 1 -- 172.21.15.22:0/1948968120 shutdown_connections 2024-03-24T09:45:55.130 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.128+0000 7f0542f94700 1 -- 172.21.15.22:0/1948968120 wait complete. 2024-03-24T09:45:55.130 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.128+0000 7f0542f94700 1 Processor -- start 2024-03-24T09:45:55.130 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.128+0000 7f0542f94700 1 -- start start 2024-03-24T09:45:55.130 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.128+0000 7f0542f94700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f053c101c90 0x7f053c0ff8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:55.130 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.128+0000 7f0542f94700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f053c1025b0 0x7f053c0ffe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:55.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.128+0000 7f0542f94700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053c100370 con 0x7f053c101c90 2024-03-24T09:45:55.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.128+0000 7f0542f94700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053c1004b0 con 0x7f053c1025b0 2024-03-24T09:45:55.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.129+0000 7f0540d30700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f053c101c90 0x7f053c0ff8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:55.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.129+0000 7f0540d30700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f053c101c90 0x7f053c0ff8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:35708/0 (socket says 172.21.15.22:35708) 2024-03-24T09:45:55.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.129+0000 7f0540d30700 1 -- 172.21.15.22:0/3660160313 learned_addr learned my addr 172.21.15.22:0/3660160313 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:45:55.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.129+0000 7f0540d30700 1 -- 172.21.15.22:0/3660160313 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f053c1025b0 msgr2=0x7f053c0ffe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:55.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.129+0000 7f053bfff700 1 --2- 172.21.15.22:0/3660160313 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f053c1025b0 0x7f053c0ffe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:55.131 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.129+0000 7f0540d30700 1 --2- 172.21.15.22:0/3660160313 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f053c1025b0 0x7f053c0ffe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:55.132 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.129+0000 7f0540d30700 1 -- 172.21.15.22:0/3660160313 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05300095f0 con 0x7f053c101c90 2024-03-24T09:45:55.132 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.129+0000 7f053bfff700 1 --2- 172.21.15.22:0/3660160313 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f053c1025b0 0x7f053c0ffe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T09:45:55.132 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.129+0000 7f0540d30700 1 --2- 172.21.15.22:0/3660160313 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f053c101c90 0x7f053c0ff8f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0530005620 tx=0x7f0530005700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:55.132 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.129+0000 7f0539ffb700 1 -- 172.21.15.22:0/3660160313 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f053001d070 con 0x7f053c101c90 2024-03-24T09:45:55.132 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.129+0000 7f0542f94700 1 -- 172.21.15.22:0/3660160313 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f053c100730 con 0x7f053c101c90 2024-03-24T09:45:55.132 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.130+0000 7f0539ffb700 1 -- 172.21.15.22:0/3660160313 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f053000f460 con 0x7f053c101c90 2024-03-24T09:45:55.132 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.130+0000 7f0539ffb700 1 -- 172.21.15.22:0/3660160313 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0530021620 con 0x7f053c101c90 2024-03-24T09:45:55.132 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.130+0000 7f0542f94700 1 -- 172.21.15.22:0/3660160313 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f053c1a2e50 con 0x7f053c101c90 2024-03-24T09:45:55.133 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.131+0000 7f0542f94700 1 -- 172.21.15.22:0/3660160313 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f053c069590 con 0x7f053c101c90 2024-03-24T09:45:55.134 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.131+0000 7f0539ffb700 1 -- 172.21.15.22:0/3660160313 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f053000fab0 con 0x7f053c101c90 2024-03-24T09:45:55.134 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.132+0000 7f0539ffb700 1 --2- 172.21.15.22:0/3660160313 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f052c06cce0 0x7f052c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:55.134 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.132+0000 7f0539ffb700 1 -- 172.21.15.22:0/3660160313 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f053008cf20 con 0x7f053c101c90 2024-03-24T09:45:55.134 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.132+0000 7f053bfff700 1 --2- 172.21.15.22:0/3660160313 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f052c06cce0 0x7f052c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:55.134 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.132+0000 7f053bfff700 1 --2- 172.21.15.22:0/3660160313 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f052c06cce0 0x7f052c06f1a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0528005fd0 tx=0x7f0528005f00 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:55.136 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.137+0000 7f0539ffb700 1 -- 172.21.15.22:0/3660160313 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f05300c09f0 con 0x7f053c101c90 2024-03-24T09:45:55.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.346+0000 7f0542f94700 1 -- 172.21.15.22:0/3660160313 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f053c105370 con 0x7f053c101c90 2024-03-24T09:45:55.345 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.346+0000 7f0539ffb700 1 -- 172.21.15.22:0/3660160313 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f05300576b0 con 0x7f053c101c90 2024-03-24T09:45:55.345 INFO:teuthology.orchestra.run.smithi022.stdout:38654705701 2024-03-24T09:45:55.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.348+0000 7f0542f94700 1 -- 172.21.15.22:0/3660160313 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f052c06cce0 msgr2=0x7f052c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:55.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.348+0000 7f0542f94700 1 --2- 172.21.15.22:0/3660160313 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f052c06cce0 0x7f052c06f1a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0528005fd0 tx=0x7f0528005f00 comp rx=0 tx=0).stop 2024-03-24T09:45:55.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.348+0000 7f0542f94700 1 -- 172.21.15.22:0/3660160313 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f053c101c90 msgr2=0x7f053c0ff8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:55.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.348+0000 7f0542f94700 1 --2- 172.21.15.22:0/3660160313 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f053c101c90 0x7f053c0ff8f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0530005620 tx=0x7f0530005700 comp rx=0 tx=0).stop 2024-03-24T09:45:55.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.348+0000 7f0542f94700 1 -- 172.21.15.22:0/3660160313 shutdown_connections 2024-03-24T09:45:55.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.348+0000 7f0542f94700 1 --2- 172.21.15.22:0/3660160313 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f052c06cce0 0x7f052c06f1a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:55.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.348+0000 7f0542f94700 1 --2- 172.21.15.22:0/3660160313 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f053c101c90 0x7f053c0ff8f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:55.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.348+0000 7f0542f94700 1 --2- 172.21.15.22:0/3660160313 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f053c1025b0 0x7f053c0ffe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:55.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.348+0000 7f0542f94700 1 -- 172.21.15.22:0/3660160313 >> 172.21.15.22:0/3660160313 conn(0x7f053c0fb480 msgr2=0x7f053c102f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:45:55.348 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.349+0000 7f0542f94700 1 -- 172.21.15.22:0/3660160313 shutdown_connections 2024-03-24T09:45:55.349 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:55.349+0000 7f0542f94700 1 -- 172.21.15.22:0/3660160313 wait complete. 2024-03-24T09:45:55.681 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:45:55.872 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:55 smithi022 ceph-mon[29512]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:55.872 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:55 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/3660160313' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T09:45:55.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:55 smithi155 ceph-mon[37016]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:55.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:55 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/3660160313' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T09:45:56.054 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.052+0000 7fb395930700 1 -- 172.21.15.22:0/524090745 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb390073aa0 msgr2=0x7fb39010cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:56.054 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.052+0000 7fb395930700 1 --2- 172.21.15.22:0/524090745 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb390073aa0 0x7fb39010cdd0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb378009910 tx=0x7fb378009c20 comp rx=0 tx=0).stop 2024-03-24T09:45:56.054 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.052+0000 7fb395930700 1 -- 172.21.15.22:0/524090745 shutdown_connections 2024-03-24T09:45:56.054 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.052+0000 7fb395930700 1 --2- 172.21.15.22:0/524090745 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb390073aa0 0x7fb39010cdd0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:56.054 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.052+0000 7fb395930700 1 --2- 172.21.15.22:0/524090745 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb390073180 0x7fb390073560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:56.054 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.052+0000 7fb395930700 1 -- 172.21.15.22:0/524090745 >> 172.21.15.22:0/524090745 conn(0x7fb3900fc5a0 msgr2=0x7fb3900fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:45:56.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.052+0000 7fb395930700 1 -- 172.21.15.22:0/524090745 shutdown_connections 2024-03-24T09:45:56.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.052+0000 7fb395930700 1 -- 172.21.15.22:0/524090745 wait complete. 2024-03-24T09:45:56.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.053+0000 7fb395930700 1 Processor -- start 2024-03-24T09:45:56.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.053+0000 7fb395930700 1 -- start start 2024-03-24T09:45:56.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.053+0000 7fb395930700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb390073180 0x7fb39019e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:56.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.053+0000 7fb395930700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb390073aa0 0x7fb39019ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:56.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.053+0000 7fb395930700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39019f450 con 0x7fb390073aa0 2024-03-24T09:45:56.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.053+0000 7fb395930700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb390198a30 con 0x7fb390073180 2024-03-24T09:45:56.055 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.053+0000 7fb38e7fc700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb390073aa0 0x7fb39019ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:56.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.053+0000 7fb38e7fc700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb390073aa0 0x7fb39019ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:35732/0 (socket says 172.21.15.22:35732) 2024-03-24T09:45:56.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.053+0000 7fb38e7fc700 1 -- 172.21.15.22:0/4130687935 learned_addr learned my addr 172.21.15.22:0/4130687935 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:45:56.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.053+0000 7fb38effd700 1 --2- 172.21.15.22:0/4130687935 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb390073180 0x7fb39019e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:56.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.054+0000 7fb38e7fc700 1 -- 172.21.15.22:0/4130687935 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb390073180 msgr2=0x7fb39019e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:56.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.054+0000 7fb38e7fc700 1 --2- 172.21.15.22:0/4130687935 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb390073180 0x7fb39019e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:56.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.054+0000 7fb38e7fc700 1 -- 172.21.15.22:0/4130687935 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3780095f0 con 0x7fb390073aa0 2024-03-24T09:45:56.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.054+0000 7fb38e7fc700 1 --2- 172.21.15.22:0/4130687935 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb390073aa0 0x7fb39019ee30 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb378006010 tx=0x7fb37800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:56.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.054+0000 7fb39492e700 1 -- 172.21.15.22:0/4130687935 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb37801d070 con 0x7fb390073aa0 2024-03-24T09:45:56.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.054+0000 7fb395930700 1 -- 172.21.15.22:0/4130687935 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb390198cb0 con 0x7fb390073aa0 2024-03-24T09:45:56.056 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.054+0000 7fb39492e700 1 -- 172.21.15.22:0/4130687935 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb3780056e0 con 0x7fb390073aa0 2024-03-24T09:45:56.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.054+0000 7fb39492e700 1 -- 172.21.15.22:0/4130687935 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb378017400 con 0x7fb390073aa0 2024-03-24T09:45:56.057 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.054+0000 7fb395930700 1 -- 172.21.15.22:0/4130687935 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3901991a0 con 0x7fb390073aa0 2024-03-24T09:45:56.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.055+0000 7fb395930700 1 -- 172.21.15.22:0/4130687935 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3901076e0 con 0x7fb390073aa0 2024-03-24T09:45:56.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.056+0000 7fb39492e700 1 -- 172.21.15.22:0/4130687935 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7fb37800f6f0 con 0x7fb390073aa0 2024-03-24T09:45:56.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.056+0000 7fb39492e700 1 --2- 172.21.15.22:0/4130687935 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fb37c06cce0 0x7fb37c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:56.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.056+0000 7fb39492e700 1 -- 172.21.15.22:0/4130687935 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fb37808d230 con 0x7fb390073aa0 2024-03-24T09:45:56.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.056+0000 7fb38effd700 1 --2- 172.21.15.22:0/4130687935 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fb37c06cce0 0x7fb37c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:56.058 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.056+0000 7fb38effd700 1 --2- 172.21.15.22:0/4130687935 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fb37c06cce0 0x7fb37c06f1a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb380009d10 tx=0x7fb380009040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:56.060 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.061+0000 7fb39492e700 1 -- 172.21.15.22:0/4130687935 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb37805b450 con 0x7fb390073aa0 2024-03-24T09:45:56.264 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.265+0000 7fb395930700 1 -- 172.21.15.22:0/4130687935 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fb390074830 con 0x7fb390073aa0 2024-03-24T09:45:56.265 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.265+0000 7fb39492e700 1 -- 172.21.15.22:0/4130687935 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fb37805afe0 con 0x7fb390073aa0 2024-03-24T09:45:56.265 INFO:teuthology.orchestra.run.smithi022.stdout:111669149716 2024-03-24T09:45:56.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.267+0000 7fb395930700 1 -- 172.21.15.22:0/4130687935 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fb37c06cce0 msgr2=0x7fb37c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:56.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.267+0000 7fb395930700 1 --2- 172.21.15.22:0/4130687935 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fb37c06cce0 0x7fb37c06f1a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb380009d10 tx=0x7fb380009040 comp rx=0 tx=0).stop 2024-03-24T09:45:56.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.267+0000 7fb395930700 1 -- 172.21.15.22:0/4130687935 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb390073aa0 msgr2=0x7fb39019ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:56.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.267+0000 7fb395930700 1 --2- 172.21.15.22:0/4130687935 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb390073aa0 0x7fb39019ee30 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb378006010 tx=0x7fb37800be20 comp rx=0 tx=0).stop 2024-03-24T09:45:56.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.267+0000 7fb395930700 1 -- 172.21.15.22:0/4130687935 shutdown_connections 2024-03-24T09:45:56.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.267+0000 7fb395930700 1 --2- 172.21.15.22:0/4130687935 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fb37c06cce0 0x7fb37c06f1a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:56.267 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.267+0000 7fb395930700 1 --2- 172.21.15.22:0/4130687935 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb390073180 0x7fb39019e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:56.268 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.267+0000 7fb395930700 1 --2- 172.21.15.22:0/4130687935 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb390073aa0 0x7fb39019ee30 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:56.268 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.267+0000 7fb395930700 1 -- 172.21.15.22:0/4130687935 >> 172.21.15.22:0/4130687935 conn(0x7fb3900fc5a0 msgr2=0x7fb3901052b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:45:56.268 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.267+0000 7fb395930700 1 -- 172.21.15.22:0/4130687935 shutdown_connections 2024-03-24T09:45:56.268 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.267+0000 7fb395930700 1 -- 172.21.15.22:0/4130687935 wait complete. 2024-03-24T09:45:56.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.315+0000 7fec0c620700 1 -- 172.21.15.22:0/2165610389 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fec04102630 msgr2=0x7fec0410ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:56.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.315+0000 7fec0c620700 1 --2- 172.21.15.22:0/2165610389 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fec04102630 0x7fec0410ab20 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7febf4009910 tx=0x7febf4009c20 comp rx=0 tx=0).stop 2024-03-24T09:45:56.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.315+0000 7fec0c620700 1 -- 172.21.15.22:0/2165610389 shutdown_connections 2024-03-24T09:45:56.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.315+0000 7fec0c620700 1 --2- 172.21.15.22:0/2165610389 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fec04102630 0x7fec0410ab20 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:56.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.315+0000 7fec0c620700 1 --2- 172.21.15.22:0/2165610389 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec04101d10 0x7fec041020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:56.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.315+0000 7fec0c620700 1 -- 172.21.15.22:0/2165610389 >> 172.21.15.22:0/2165610389 conn(0x7fec040fb500 msgr2=0x7fec040fd920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:45:56.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.315+0000 7fec0c620700 1 -- 172.21.15.22:0/2165610389 shutdown_connections 2024-03-24T09:45:56.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.315+0000 7fec0c620700 1 -- 172.21.15.22:0/2165610389 wait complete. 2024-03-24T09:45:56.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.316+0000 7fec0c620700 1 Processor -- start 2024-03-24T09:45:56.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.316+0000 7fec0c620700 1 -- start start 2024-03-24T09:45:56.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.316+0000 7fec0c620700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fec04101d10 0x7fec040ff970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:56.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.316+0000 7fec0c620700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec04102630 0x7fec040ffeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:56.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.316+0000 7fec0c620700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec041003f0 con 0x7fec04101d10 2024-03-24T09:45:56.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.316+0000 7fec0c620700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec04100530 con 0x7fec04102630 2024-03-24T09:45:56.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.316+0000 7fec0a3bc700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fec04101d10 0x7fec040ff970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:56.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.316+0000 7fec0a3bc700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fec04101d10 0x7fec040ff970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:35760/0 (socket says 172.21.15.22:35760) 2024-03-24T09:45:56.538 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.316+0000 7fec0a3bc700 1 -- 172.21.15.22:0/420100003 learned_addr learned my addr 172.21.15.22:0/420100003 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:45:56.539 INFO:teuthology.orchestra.run.smithi022.stdout:60129542177 2024-03-24T09:45:56.539 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.316+0000 7fec09bbb700 1 --2- 172.21.15.22:0/420100003 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec04102630 0x7fec040ffeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:56.539 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.316+0000 7fec0a3bc700 1 -- 172.21.15.22:0/420100003 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec04102630 msgr2=0x7fec040ffeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:56.540 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.316+0000 7fec0a3bc700 1 --2- 172.21.15.22:0/420100003 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec04102630 0x7fec040ffeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:56.540 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.316+0000 7fec0a3bc700 1 -- 172.21.15.22:0/420100003 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf40095f0 con 0x7fec04101d10 2024-03-24T09:45:56.540 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.316+0000 7fec09bbb700 1 --2- 172.21.15.22:0/420100003 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec04102630 0x7fec040ffeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:45:56.540 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.317+0000 7fec0a3bc700 1 --2- 172.21.15.22:0/420100003 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fec04101d10 0x7fec040ff970 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fec0000d900 tx=0x7fec0000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:56.540 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.317+0000 7febfb7fe700 1 -- 172.21.15.22:0/420100003 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec00004a00 con 0x7fec04101d10 2024-03-24T09:45:56.540 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.317+0000 7fec0c620700 1 -- 172.21.15.22:0/420100003 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec041a2bf0 con 0x7fec04101d10 2024-03-24T09:45:56.540 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.317+0000 7febfb7fe700 1 -- 172.21.15.22:0/420100003 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fec000054e0 con 0x7fec04101d10 2024-03-24T09:45:56.540 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.317+0000 7febfb7fe700 1 -- 172.21.15.22:0/420100003 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec00009d60 con 0x7fec04101d10 2024-03-24T09:45:56.540 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.317+0000 7fec0c620700 1 -- 172.21.15.22:0/420100003 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec041a2f50 con 0x7fec04101d10 2024-03-24T09:45:56.540 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.318+0000 7fec0c620700 1 -- 172.21.15.22:0/420100003 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec041082c0 con 0x7fec04101d10 2024-03-24T09:45:56.541 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.318+0000 7febfb7fe700 1 -- 172.21.15.22:0/420100003 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7fec00005020 con 0x7fec04101d10 2024-03-24T09:45:56.541 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.319+0000 7febfb7fe700 1 --2- 172.21.15.22:0/420100003 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7febf006cce0 0x7febf006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:45:56.541 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.319+0000 7febfb7fe700 1 -- 172.21.15.22:0/420100003 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fec0008c0d0 con 0x7fec04101d10 2024-03-24T09:45:56.541 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.319+0000 7fec09bbb700 1 --2- 172.21.15.22:0/420100003 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7febf006cce0 0x7febf006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:45:56.541 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.319+0000 7fec09bbb700 1 --2- 172.21.15.22:0/420100003 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7febf006cce0 0x7febf006f1a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fec041010a0 tx=0x7febf401a040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:45:56.541 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.324+0000 7febfb7fe700 1 -- 172.21.15.22:0/420100003 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fec000567e0 con 0x7fec04101d10 2024-03-24T09:45:56.541 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.531+0000 7fec0c620700 1 -- 172.21.15.22:0/420100003 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fec04069810 con 0x7fec04101d10 2024-03-24T09:45:56.541 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.532+0000 7febfb7fe700 1 -- 172.21.15.22:0/420100003 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fec00059e00 con 0x7fec04101d10 2024-03-24T09:45:56.541 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.534+0000 7fec0c620700 1 -- 172.21.15.22:0/420100003 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7febf006cce0 msgr2=0x7febf006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:56.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.534+0000 7fec0c620700 1 --2- 172.21.15.22:0/420100003 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7febf006cce0 0x7febf006f1a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fec041010a0 tx=0x7febf401a040 comp rx=0 tx=0).stop 2024-03-24T09:45:56.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.534+0000 7fec0c620700 1 -- 172.21.15.22:0/420100003 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fec04101d10 msgr2=0x7fec040ff970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:45:56.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.534+0000 7fec0c620700 1 --2- 172.21.15.22:0/420100003 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fec04101d10 0x7fec040ff970 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fec0000d900 tx=0x7fec0000dcc0 comp rx=0 tx=0).stop 2024-03-24T09:45:56.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.534+0000 7fec0c620700 1 -- 172.21.15.22:0/420100003 shutdown_connections 2024-03-24T09:45:56.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.534+0000 7fec0c620700 1 --2- 172.21.15.22:0/420100003 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7febf006cce0 0x7febf006f1a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:56.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.534+0000 7fec0c620700 1 --2- 172.21.15.22:0/420100003 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fec04101d10 0x7fec040ff970 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:56.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.534+0000 7fec0c620700 1 --2- 172.21.15.22:0/420100003 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec04102630 0x7fec040ffeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:45:56.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.534+0000 7fec0c620700 1 -- 172.21.15.22:0/420100003 >> 172.21.15.22:0/420100003 conn(0x7fec040fb500 msgr2=0x7fec04105400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:45:56.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.534+0000 7fec0c620700 1 -- 172.21.15.22:0/420100003 shutdown_connections 2024-03-24T09:45:56.542 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:45:56.534+0000 7fec0c620700 1 -- 172.21.15.22:0/420100003 wait complete. 2024-03-24T09:45:56.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:56 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/4130687935' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T09:45:56.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:56 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/420100003' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T09:45:56.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:56 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/4130687935' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T09:45:56.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:56 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/420100003' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T09:45:57.752 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:57 smithi022 ceph-mon[29512]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:57.779 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149713 got 111669149716 for osd.3 2024-03-24T09:45:57.779 DEBUG:teuthology.parallel:result is None 2024-03-24T09:45:57.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:57 smithi155 ceph-mon[37016]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:58.861 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542175 got 60129542177 for osd.1 2024-03-24T09:45:58.861 DEBUG:teuthology.parallel:result is None 2024-03-24T09:45:59.853 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:45:59 smithi022 ceph-mon[29512]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:45:59.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:45:59 smithi155 ceph-mon[37016]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:46:00.152 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705699 got 38654705701 for osd.0 2024-03-24T09:46:00.153 DEBUG:teuthology.parallel:result is None 2024-03-24T09:46:00.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.560+0000 7f9212c84700 1 -- 172.21.15.22:0/1907232425 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f920c105950 msgr2=0x7f920c105d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:00.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.560+0000 7f9212c84700 1 --2- 172.21.15.22:0/1907232425 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f920c105950 0x7f920c105d30 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9200009910 tx=0x7f9200009c20 comp rx=0 tx=0).stop 2024-03-24T09:46:00.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.560+0000 7f9212c84700 1 -- 172.21.15.22:0/1907232425 shutdown_connections 2024-03-24T09:46:00.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.560+0000 7f9212c84700 1 --2- 172.21.15.22:0/1907232425 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f920c0ff950 0x7f920c0ffdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:00.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.560+0000 7f9212c84700 1 --2- 172.21.15.22:0/1907232425 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f920c105950 0x7f920c105d30 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:00.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.560+0000 7f9212c84700 1 -- 172.21.15.22:0/1907232425 >> 172.21.15.22:0/1907232425 conn(0x7f920c076640 msgr2=0x7f920c0fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:00.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.560+0000 7f9212c84700 1 -- 172.21.15.22:0/1907232425 shutdown_connections 2024-03-24T09:46:00.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.560+0000 7f9212c84700 1 -- 172.21.15.22:0/1907232425 wait complete. 2024-03-24T09:46:00.564 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.561+0000 7f9212c84700 1 Processor -- start 2024-03-24T09:46:00.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.561+0000 7f9212c84700 1 -- start start 2024-03-24T09:46:00.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.561+0000 7f9212c84700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f920c0ff950 0x7f920c19b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:00.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.561+0000 7f9212c84700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f920c105950 0x7f920c19bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:00.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.561+0000 7f9212c84700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f920c19c2e0 con 0x7f920c0ff950 2024-03-24T09:46:00.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.561+0000 7f9212c84700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f920c195850 con 0x7f920c105950 2024-03-24T09:46:00.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.561+0000 7f9210a20700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f920c0ff950 0x7f920c19b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:00.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.561+0000 7f9210a20700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f920c0ff950 0x7f920c19b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:54626/0 (socket says 172.21.15.22:54626) 2024-03-24T09:46:00.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.561+0000 7f9210a20700 1 -- 172.21.15.22:0/986355586 learned_addr learned my addr 172.21.15.22:0/986355586 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:46:00.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.561+0000 7f920bfff700 1 --2- 172.21.15.22:0/986355586 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f920c105950 0x7f920c19bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:00.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.561+0000 7f9210a20700 1 -- 172.21.15.22:0/986355586 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f920c105950 msgr2=0x7f920c19bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:00.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.561+0000 7f9210a20700 1 --2- 172.21.15.22:0/986355586 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f920c105950 0x7f920c19bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:00.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.561+0000 7f9210a20700 1 -- 172.21.15.22:0/986355586 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92000095f0 con 0x7f920c0ff950 2024-03-24T09:46:00.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.562+0000 7f9210a20700 1 --2- 172.21.15.22:0/986355586 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f920c0ff950 0x7f920c19b710 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f920000bd70 tx=0x7f920000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:00.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.562+0000 7f9209ffb700 1 -- 172.21.15.22:0/986355586 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f920001d070 con 0x7f920c0ff950 2024-03-24T09:46:00.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.562+0000 7f9209ffb700 1 -- 172.21.15.22:0/986355586 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f920000f460 con 0x7f920c0ff950 2024-03-24T09:46:00.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.562+0000 7f9212c84700 1 -- 172.21.15.22:0/986355586 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f920c195ad0 con 0x7f920c0ff950 2024-03-24T09:46:00.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.562+0000 7f9209ffb700 1 -- 172.21.15.22:0/986355586 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9200017630 con 0x7f920c0ff950 2024-03-24T09:46:00.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.562+0000 7f9212c84700 1 -- 172.21.15.22:0/986355586 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f920c195fc0 con 0x7f920c0ff950 2024-03-24T09:46:00.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.563+0000 7f9212c84700 1 -- 172.21.15.22:0/986355586 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f920c107f50 con 0x7f920c0ff950 2024-03-24T09:46:00.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.564+0000 7f9209ffb700 1 -- 172.21.15.22:0/986355586 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f920000f5d0 con 0x7f920c0ff950 2024-03-24T09:46:00.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.564+0000 7f9209ffb700 1 --2- 172.21.15.22:0/986355586 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f91fc06cce0 0x7f91fc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:00.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.564+0000 7f9209ffb700 1 -- 172.21.15.22:0/986355586 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f920008d380 con 0x7f920c0ff950 2024-03-24T09:46:00.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.564+0000 7f920bfff700 1 --2- 172.21.15.22:0/986355586 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f91fc06cce0 0x7f91fc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:00.567 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.564+0000 7f920bfff700 1 --2- 172.21.15.22:0/986355586 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f91fc06cce0 0x7f91fc06f1a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f91f8006fd0 tx=0x7f91f8008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:00.569 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.569+0000 7f9209ffb700 1 -- 172.21.15.22:0/986355586 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f920005b5a0 con 0x7f920c0ff950 2024-03-24T09:46:00.774 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.773+0000 7f9212c84700 1 -- 172.21.15.22:0/986355586 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f920c101090 con 0x7f920c0ff950 2024-03-24T09:46:00.774 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.774+0000 7f9209ffb700 1 -- 172.21.15.22:0/986355586 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f920005b130 con 0x7f920c0ff950 2024-03-24T09:46:00.775 INFO:teuthology.orchestra.run.smithi022.stdout:146028888082 2024-03-24T09:46:00.776 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.776+0000 7f9212c84700 1 -- 172.21.15.22:0/986355586 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f91fc06cce0 msgr2=0x7f91fc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:00.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.776+0000 7f9212c84700 1 --2- 172.21.15.22:0/986355586 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f91fc06cce0 0x7f91fc06f1a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f91f8006fd0 tx=0x7f91f8008040 comp rx=0 tx=0).stop 2024-03-24T09:46:00.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.776+0000 7f9212c84700 1 -- 172.21.15.22:0/986355586 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f920c0ff950 msgr2=0x7f920c19b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:00.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.776+0000 7f9212c84700 1 --2- 172.21.15.22:0/986355586 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f920c0ff950 0x7f920c19b710 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f920000bd70 tx=0x7f920000be50 comp rx=0 tx=0).stop 2024-03-24T09:46:00.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.776+0000 7f9212c84700 1 -- 172.21.15.22:0/986355586 shutdown_connections 2024-03-24T09:46:00.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.776+0000 7f9212c84700 1 --2- 172.21.15.22:0/986355586 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f91fc06cce0 0x7f91fc06f1a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:00.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.776+0000 7f9212c84700 1 --2- 172.21.15.22:0/986355586 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f920c0ff950 0x7f920c19b710 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:00.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.776+0000 7f9212c84700 1 --2- 172.21.15.22:0/986355586 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f920c105950 0x7f920c19bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:00.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.776+0000 7f9212c84700 1 -- 172.21.15.22:0/986355586 >> 172.21.15.22:0/986355586 conn(0x7f920c076640 msgr2=0x7f920c0fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:00.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.776+0000 7f9212c84700 1 -- 172.21.15.22:0/986355586 shutdown_connections 2024-03-24T09:46:00.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.776+0000 7f9212c84700 1 -- 172.21.15.22:0/986355586 wait complete. 2024-03-24T09:46:00.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.993+0000 7f90f4b4b700 1 -- 172.21.15.22:0/1957746701 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f90f0105970 msgr2=0x7f90f0105d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:00.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.993+0000 7f90f4b4b700 1 --2- 172.21.15.22:0/1957746701 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f90f0105970 0x7f90f0105d50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f90e0009910 tx=0x7f90e0009c20 comp rx=0 tx=0).stop 2024-03-24T09:46:00.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.993+0000 7f90f4b4b700 1 -- 172.21.15.22:0/1957746701 shutdown_connections 2024-03-24T09:46:00.995 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.993+0000 7f90f4b4b700 1 --2- 172.21.15.22:0/1957746701 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90f00ff970 0x7f90f00ffdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:00.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.993+0000 7f90f4b4b700 1 --2- 172.21.15.22:0/1957746701 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f90f0105970 0x7f90f0105d50 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:00.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.993+0000 7f90f4b4b700 1 -- 172.21.15.22:0/1957746701 >> 172.21.15.22:0/1957746701 conn(0x7f90f00fb480 msgr2=0x7f90f00fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:00.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.993+0000 7f90f4b4b700 1 -- 172.21.15.22:0/1957746701 shutdown_connections 2024-03-24T09:46:00.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.993+0000 7f90f4b4b700 1 -- 172.21.15.22:0/1957746701 wait complete. 2024-03-24T09:46:00.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.993+0000 7f90f4b4b700 1 Processor -- start 2024-03-24T09:46:00.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.994+0000 7f90f4b4b700 1 -- start start 2024-03-24T09:46:00.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.994+0000 7f90f4b4b700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f90f00ff970 0x7f90f0197850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:00.996 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.994+0000 7f90f4b4b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90f0105970 0x7f90f0197d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:00.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.994+0000 7f90f4b4b700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f0198360 con 0x7f90f00ff970 2024-03-24T09:46:00.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.994+0000 7f90f4b4b700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f01984a0 con 0x7f90f0105970 2024-03-24T09:46:00.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.994+0000 7f90ee59c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f90f00ff970 0x7f90f0197850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:00.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.994+0000 7f90ee59c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f90f00ff970 0x7f90f0197850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:54638/0 (socket says 172.21.15.22:54638) 2024-03-24T09:46:00.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.994+0000 7f90ee59c700 1 -- 172.21.15.22:0/3116618431 learned_addr learned my addr 172.21.15.22:0/3116618431 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:46:00.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.994+0000 7f90edd9b700 1 --2- 172.21.15.22:0/3116618431 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90f0105970 0x7f90f0197d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:00.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.994+0000 7f90ee59c700 1 -- 172.21.15.22:0/3116618431 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90f0105970 msgr2=0x7f90f0197d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:00.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.994+0000 7f90ee59c700 1 --2- 172.21.15.22:0/3116618431 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90f0105970 0x7f90f0197d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:00.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.994+0000 7f90ee59c700 1 -- 172.21.15.22:0/3116618431 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e00095f0 con 0x7f90f00ff970 2024-03-24T09:46:00.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.994+0000 7f90ee59c700 1 --2- 172.21.15.22:0/3116618431 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f90f00ff970 0x7f90f0197850 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f90e0006010 tx=0x7f90e000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:00.997 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.994+0000 7f90df7fe700 1 -- 172.21.15.22:0/3116618431 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90e001d070 con 0x7f90f00ff970 2024-03-24T09:46:00.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.994+0000 7f90df7fe700 1 -- 172.21.15.22:0/3116618431 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f90e000f460 con 0x7f90f00ff970 2024-03-24T09:46:00.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.994+0000 7f90f4b4b700 1 -- 172.21.15.22:0/3116618431 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90f01a08c0 con 0x7f90f00ff970 2024-03-24T09:46:00.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.995+0000 7f90f4b4b700 1 -- 172.21.15.22:0/3116618431 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90f01a0c20 con 0x7f90f00ff970 2024-03-24T09:46:00.998 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.995+0000 7f90df7fe700 1 -- 172.21.15.22:0/3116618431 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90e0017630 con 0x7f90f00ff970 2024-03-24T09:46:00.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.995+0000 7f90f4b4b700 1 -- 172.21.15.22:0/3116618431 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90f0107f70 con 0x7f90f00ff970 2024-03-24T09:46:00.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.996+0000 7f90df7fe700 1 -- 172.21.15.22:0/3116618431 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f90e000f5d0 con 0x7f90f00ff970 2024-03-24T09:46:00.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.996+0000 7f90df7fe700 1 --2- 172.21.15.22:0/3116618431 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f90d806cce0 0x7f90d806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:00.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.996+0000 7f90df7fe700 1 -- 172.21.15.22:0/3116618431 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f90e008d390 con 0x7f90f00ff970 2024-03-24T09:46:00.999 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.996+0000 7f90edd9b700 1 --2- 172.21.15.22:0/3116618431 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f90d806cce0 0x7f90d806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:01.000 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:00.997+0000 7f90edd9b700 1 --2- 172.21.15.22:0/3116618431 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f90d806cce0 0x7f90d806f1a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f90f0069500 tx=0x7f90e4005cf0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:01.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:01.001+0000 7f90df7fe700 1 -- 172.21.15.22:0/3116618431 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f90e0057b60 con 0x7f90f00ff970 2024-03-24T09:46:01.208 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:01.207+0000 7f90f4b4b700 1 -- 172.21.15.22:0/3116618431 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f90f01010b0 con 0x7f90f00ff970 2024-03-24T09:46:01.208 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:01.207+0000 7f90df7fe700 1 -- 172.21.15.22:0/3116618431 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f90f01010b0 con 0x7f90f00ff970 2024-03-24T09:46:01.208 INFO:teuthology.orchestra.run.smithi022.stdout:81604378652 2024-03-24T09:46:01.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:01.209+0000 7f90f4b4b700 1 -- 172.21.15.22:0/3116618431 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f90d806cce0 msgr2=0x7f90d806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:01.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:01.209+0000 7f90f4b4b700 1 --2- 172.21.15.22:0/3116618431 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f90d806cce0 0x7f90d806f1a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f90f0069500 tx=0x7f90e4005cf0 comp rx=0 tx=0).stop 2024-03-24T09:46:01.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:01.209+0000 7f90f4b4b700 1 -- 172.21.15.22:0/3116618431 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f90f00ff970 msgr2=0x7f90f0197850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:01.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:01.209+0000 7f90f4b4b700 1 --2- 172.21.15.22:0/3116618431 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f90f00ff970 0x7f90f0197850 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f90e0006010 tx=0x7f90e000be20 comp rx=0 tx=0).stop 2024-03-24T09:46:01.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:01.209+0000 7f90f4b4b700 1 -- 172.21.15.22:0/3116618431 shutdown_connections 2024-03-24T09:46:01.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:01.209+0000 7f90f4b4b700 1 --2- 172.21.15.22:0/3116618431 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f90d806cce0 0x7f90d806f1a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:01.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:01.209+0000 7f90f4b4b700 1 --2- 172.21.15.22:0/3116618431 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f90f00ff970 0x7f90f0197850 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:01.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:01.209+0000 7f90f4b4b700 1 --2- 172.21.15.22:0/3116618431 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90f0105970 0x7f90f0197d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:01.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:01.209+0000 7f90f4b4b700 1 -- 172.21.15.22:0/3116618431 >> 172.21.15.22:0/3116618431 conn(0x7f90f00fb480 msgr2=0x7f90f00fc130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:01.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:01.210+0000 7f90f4b4b700 1 -- 172.21.15.22:0/3116618431 shutdown_connections 2024-03-24T09:46:01.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:01.210+0000 7f90f4b4b700 1 -- 172.21.15.22:0/3116618431 wait complete. 2024-03-24T09:46:01.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:01 smithi155 ceph-mon[37016]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:46:01.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:01 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/986355586' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T09:46:01.477 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:01 smithi022 ceph-mon[29512]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:46:01.478 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:01 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/986355586' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T09:46:01.758 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888080 got 146028888082 for osd.4 2024-03-24T09:46:01.758 DEBUG:teuthology.parallel:result is None 2024-03-24T09:46:02.182 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378649 got 81604378652 for osd.2 2024-03-24T09:46:02.182 DEBUG:teuthology.parallel:result is None 2024-03-24T09:46:02.182 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-24T09:46:02.183 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T09:46:02.399 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:46:02.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:02 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/3116618431' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T09:46:02.514 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:02 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/3116618431' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T09:46:03.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:03 smithi155 ceph-mon[37016]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:46:03.503 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:03 smithi022 ceph-mon[29512]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:46:03.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:03.974+0000 7f7c49a6f700 1 -- 172.21.15.22:0/1496540030 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7c44100a70 msgr2=0x7f7c44100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:03.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:03.974+0000 7f7c49a6f700 1 --2- 172.21.15.22:0/1496540030 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7c44100a70 0x7f7c44100ef0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7c38009910 tx=0x7f7c38009c20 comp rx=0 tx=0).stop 2024-03-24T09:46:03.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:03.974+0000 7f7c49a6f700 1 -- 172.21.15.22:0/1496540030 shutdown_connections 2024-03-24T09:46:03.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:03.974+0000 7f7c49a6f700 1 --2- 172.21.15.22:0/1496540030 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7c44100a70 0x7f7c44100ef0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:03.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:03.974+0000 7f7c49a6f700 1 --2- 172.21.15.22:0/1496540030 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c44106a70 0x7f7c44106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:03.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:03.974+0000 7f7c49a6f700 1 -- 172.21.15.22:0/1496540030 >> 172.21.15.22:0/1496540030 conn(0x7f7c440fc580 msgr2=0x7f7c440fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:03.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:03.974+0000 7f7c49a6f700 1 -- 172.21.15.22:0/1496540030 shutdown_connections 2024-03-24T09:46:03.977 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:03.974+0000 7f7c49a6f700 1 -- 172.21.15.22:0/1496540030 wait complete. 2024-03-24T09:46:03.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:03.975+0000 7f7c49a6f700 1 Processor -- start 2024-03-24T09:46:03.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:03.975+0000 7f7c49a6f700 1 -- start start 2024-03-24T09:46:03.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:03.975+0000 7f7c49a6f700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c44100a70 0x7f7c440730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:03.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:03.975+0000 7f7c49a6f700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7c44106a70 0x7f7c440735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:03.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:03.975+0000 7f7c49a6f700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c44073b30 con 0x7f7c44106a70 2024-03-24T09:46:03.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:03.975+0000 7f7c49a6f700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c44073c70 con 0x7f7c44100a70 2024-03-24T09:46:03.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:03.975+0000 7f7c42ffd700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7c44106a70 0x7f7c440735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:03.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:03.975+0000 7f7c42ffd700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7c44106a70 0x7f7c440735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:54658/0 (socket says 172.21.15.22:54658) 2024-03-24T09:46:03.978 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:03.975+0000 7f7c42ffd700 1 -- 172.21.15.22:0/565263716 learned_addr learned my addr 172.21.15.22:0/565263716 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:46:03.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:03.975+0000 7f7c437fe700 1 --2- 172.21.15.22:0/565263716 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c44100a70 0x7f7c440730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:03.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:03.976+0000 7f7c42ffd700 1 -- 172.21.15.22:0/565263716 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c44100a70 msgr2=0x7f7c440730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:03.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:03.976+0000 7f7c42ffd700 1 --2- 172.21.15.22:0/565263716 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c44100a70 0x7f7c440730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:03.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:03.976+0000 7f7c42ffd700 1 -- 172.21.15.22:0/565263716 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c380095f0 con 0x7f7c44106a70 2024-03-24T09:46:03.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:03.976+0000 7f7c42ffd700 1 --2- 172.21.15.22:0/565263716 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7c44106a70 0x7f7c440735f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7c3800bd70 tx=0x7f7c3800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:03.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:03.976+0000 7f7c40ff9700 1 -- 172.21.15.22:0/565263716 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c3801d070 con 0x7f7c44106a70 2024-03-24T09:46:03.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:03.976+0000 7f7c49a6f700 1 -- 172.21.15.22:0/565263716 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c44073ef0 con 0x7f7c44106a70 2024-03-24T09:46:03.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:03.976+0000 7f7c40ff9700 1 -- 172.21.15.22:0/565263716 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7c3800f460 con 0x7f7c44106a70 2024-03-24T09:46:03.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:03.976+0000 7f7c40ff9700 1 -- 172.21.15.22:0/565263716 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c38017630 con 0x7f7c44106a70 2024-03-24T09:46:03.979 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:03.976+0000 7f7c49a6f700 1 -- 172.21.15.22:0/565263716 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c441a2dc0 con 0x7f7c44106a70 2024-03-24T09:46:03.980 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:03.977+0000 7f7c49a6f700 1 -- 172.21.15.22:0/565263716 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c44109090 con 0x7f7c44106a70 2024-03-24T09:46:03.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:03.977+0000 7f7c40ff9700 1 -- 172.21.15.22:0/565263716 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f7c3800f5d0 con 0x7f7c44106a70 2024-03-24T09:46:03.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:03.978+0000 7f7c40ff9700 1 --2- 172.21.15.22:0/565263716 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f7c2c06cce0 0x7f7c2c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:03.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:03.978+0000 7f7c40ff9700 1 -- 172.21.15.22:0/565263716 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f7c3808d430 con 0x7f7c44106a70 2024-03-24T09:46:03.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:03.978+0000 7f7c437fe700 1 --2- 172.21.15.22:0/565263716 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f7c2c06cce0 0x7f7c2c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:03.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:03.979+0000 7f7c437fe700 1 --2- 172.21.15.22:0/565263716 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f7c2c06cce0 0x7f7c2c06f1a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7c34005fd0 tx=0x7f7c3400a560 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:03.983 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:03.983+0000 7f7c40ff9700 1 -- 172.21.15.22:0/565263716 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7c38057b40 con 0x7f7c44106a70 2024-03-24T09:46:04.189 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:04.189+0000 7f7c49a6f700 1 -- 172.21.15.22:0/565263716 --> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7c440611f0 con 0x7f7c2c06cce0 2024-03-24T09:46:04.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:04.190+0000 7f7c40ff9700 1 -- 172.21.15.22:0/565263716 <== mgr.14215 v2:172.21.15.22:6800/773863266 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27923 (secure 0 0 0) 0x7f7c440611f0 con 0x7f7c2c06cce0 2024-03-24T09:46:04.191 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:46:04.193 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:04.192+0000 7f7c49a6f700 1 -- 172.21.15.22:0/565263716 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f7c2c06cce0 msgr2=0x7f7c2c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:04.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:04.192+0000 7f7c49a6f700 1 --2- 172.21.15.22:0/565263716 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f7c2c06cce0 0x7f7c2c06f1a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7c34005fd0 tx=0x7f7c3400a560 comp rx=0 tx=0).stop 2024-03-24T09:46:04.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:04.192+0000 7f7c49a6f700 1 -- 172.21.15.22:0/565263716 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7c44106a70 msgr2=0x7f7c440735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:04.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:04.192+0000 7f7c49a6f700 1 --2- 172.21.15.22:0/565263716 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7c44106a70 0x7f7c440735f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7c3800bd70 tx=0x7f7c3800be50 comp rx=0 tx=0).stop 2024-03-24T09:46:04.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:04.192+0000 7f7c49a6f700 1 -- 172.21.15.22:0/565263716 shutdown_connections 2024-03-24T09:46:04.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:04.192+0000 7f7c49a6f700 1 --2- 172.21.15.22:0/565263716 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f7c2c06cce0 0x7f7c2c06f1a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:04.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:04.192+0000 7f7c49a6f700 1 --2- 172.21.15.22:0/565263716 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c44100a70 0x7f7c440730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:04.194 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:04.192+0000 7f7c49a6f700 1 --2- 172.21.15.22:0/565263716 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7c44106a70 0x7f7c440735f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:04.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:04.192+0000 7f7c49a6f700 1 -- 172.21.15.22:0/565263716 >> 172.21.15.22:0/565263716 conn(0x7f7c440fc580 msgr2=0x7f7c440fd210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:04.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:04.193+0000 7f7c49a6f700 1 -- 172.21.15.22:0/565263716 shutdown_connections 2024-03-24T09:46:04.195 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:04.193+0000 7f7c49a6f700 1 -- 172.21.15.22:0/565263716 wait complete. 2024-03-24T09:46:04.195 INFO:teuthology.orchestra.run.smithi022.stderr:dumped all 2024-03-24T09:46:04.992 INFO:teuthology.orchestra.run.smithi022.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2024-03-24T09:46:02.497885+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":166008,"kb_used_data":3876,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":562304904,"statfs":{"total":575970213888,"available":575800221696,"internally_reserved":0,"allocated":3969024,"data_stored":2609166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":165927590},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002097"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":186,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T09:45:12.572979+0000","last_change":"2024-03-24T09:44:46.554917+0000","last_active":"2024-03-24T09:45:12.572979+0000","last_peered":"2024-03-24T09:45:12.572979+0000","last_clean":"2024-03-24T09:45:12.572979+0000","last_became_active":"2024-03-24T09:44:46.554634+0000","last_became_peered":"2024-03-24T09:44:46.554634+0000","last_unstale":"2024-03-24T09:45:12.572979+0000","last_undegraded":"2024-03-24T09:45:12.572979+0000","last_fullsized":"2024-03-24T09:45:12.572979+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:43:57.536831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:43:57.536831+0000","last_clean_scrub_stamp":"2024-03-24T09:43:57.536831+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:46:54.625430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27376,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717776,"statfs":{"total":95995035648,"available":95967002624,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]}]},{"osd":4,"up_from":34,"seq":146028888082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27376,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717776,"statfs":{"total":95995035648,"available":95967002624,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 09:45:48 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.46800000000000003}]},{"osd":1,"last update":"Sun Mar 24 09:45:48 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.497}]},{"osd":2,"last update":"Sun Mar 24 09:45:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.41699999999999998}]},{"osd":3,"last update":"Sun Mar 24 09:45:48 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.437},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.35499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149718,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 09:45:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.60399999999999998}]},{"osd":1,"last update":"Sun Mar 24 09:45:23 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.63500000000000001}]},{"osd":2,"last update":"Sun Mar 24 09:45:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.61899999999999999}]},{"osd":4,"last update":"Sun Mar 24 09:45:48 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.46400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]}]},{"osd":2,"up_from":19,"seq":81604378652,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27376,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717776,"statfs":{"total":95995035648,"available":95967002624,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 09:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"last":0.31900000000000001}]},{"osd":1,"last update":"Sun Mar 24 09:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.35299999999999998}]},{"osd":3,"last update":"Sun Mar 24 09:45:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.433},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.52200000000000002}]},{"osd":4,"last update":"Sun Mar 24 09:45:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.48399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]}]},{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Mar 24 09:45:36 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.32300000000000001,"15min":0.32200000000000001},"min":{"1min":0.224,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.44700000000000001,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.33100000000000002,"15min":0.33000000000000002},"min":{"1min":0.23400000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.46300000000000002,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.52300000000000002}]},{"osd":2,"last update":"Sun Mar 24 09:44:59 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.37,"15min":0.37},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.54000000000000004}]},{"osd":3,"last update":"Sun Mar 24 09:45:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.69799999999999995}]},{"osd":4,"last update":"Sun Mar 24 09:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":6.141,"5min":6.141,"15min":6.141},"last":0.623},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.67100000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]}]},{"osd":1,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 09:45:33 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.33600000000000002,"15min":0.33000000000000002},"min":{"1min":0.255,"5min":0.216,"15min":0.216},"max":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.34100000000000003,"15min":0.33700000000000002},"min":{"1min":0.27900000000000003,"5min":0.219,"15min":0.219},"max":{"1min":0.50800000000000001,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.308}]},{"osd":2,"last update":"Sun Mar 24 09:45:00 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.373}]},{"osd":3,"last update":"Sun Mar 24 09:45:23 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.65500000000000003}]},{"osd":4,"last update":"Sun Mar 24 09:45:47 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.70299999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T09:46:04.994 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T09:46:05.207 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:46:05.637 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:05 smithi022 ceph-mon[29512]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T09:46:05.637 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:05 smithi022 ceph-mon[29512]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:46:05.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:05 smithi155 ceph-mon[37016]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T09:46:05.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:05 smithi155 ceph-mon[37016]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:46:06.781 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.778+0000 7f6d5311c700 1 -- 172.21.15.22:0/3922335933 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6d4c102570 msgr2=0x7f6d4c10aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:06.781 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.778+0000 7f6d5311c700 1 --2- 172.21.15.22:0/3922335933 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6d4c102570 0x7f6d4c10aa60 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6d38009910 tx=0x7f6d38009c20 comp rx=0 tx=0).stop 2024-03-24T09:46:06.781 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.778+0000 7f6d5311c700 1 -- 172.21.15.22:0/3922335933 shutdown_connections 2024-03-24T09:46:06.781 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.778+0000 7f6d5311c700 1 --2- 172.21.15.22:0/3922335933 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6d4c102570 0x7f6d4c10aa60 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:06.781 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.778+0000 7f6d5311c700 1 --2- 172.21.15.22:0/3922335933 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d4c101c50 0x7f6d4c102030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:06.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.778+0000 7f6d5311c700 1 -- 172.21.15.22:0/3922335933 >> 172.21.15.22:0/3922335933 conn(0x7f6d4c076640 msgr2=0x7f6d4c0fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:06.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.778+0000 7f6d5311c700 1 -- 172.21.15.22:0/3922335933 shutdown_connections 2024-03-24T09:46:06.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.778+0000 7f6d5311c700 1 -- 172.21.15.22:0/3922335933 wait complete. 2024-03-24T09:46:06.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.778+0000 7f6d5311c700 1 Processor -- start 2024-03-24T09:46:06.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.778+0000 7f6d5311c700 1 -- start start 2024-03-24T09:46:06.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.779+0000 7f6d5311c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6d4c101c50 0x7f6d4c19c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:06.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.779+0000 7f6d5311c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d4c102570 0x7f6d4c19cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:06.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.779+0000 7f6d5311c700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4c19d290 con 0x7f6d4c101c50 2024-03-24T09:46:06.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.779+0000 7f6d5311c700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4c196800 con 0x7f6d4c102570 2024-03-24T09:46:06.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.779+0000 7f6d50eb8700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6d4c101c50 0x7f6d4c19c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:06.783 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.779+0000 7f6d50eb8700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6d4c101c50 0x7f6d4c19c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:54688/0 (socket says 172.21.15.22:54688) 2024-03-24T09:46:06.783 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.779+0000 7f6d50eb8700 1 -- 172.21.15.22:0/1790327949 learned_addr learned my addr 172.21.15.22:0/1790327949 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:46:06.783 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.779+0000 7f6d50eb8700 1 -- 172.21.15.22:0/1790327949 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d4c102570 msgr2=0x7f6d4c19cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:06.783 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.779+0000 7f6d50eb8700 1 --2- 172.21.15.22:0/1790327949 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d4c102570 0x7f6d4c19cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:06.783 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.779+0000 7f6d50eb8700 1 -- 172.21.15.22:0/1790327949 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d380095f0 con 0x7f6d4c101c50 2024-03-24T09:46:06.783 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.779+0000 7f6d50eb8700 1 --2- 172.21.15.22:0/1790327949 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6d4c101c50 0x7f6d4c19c6c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6d4000d8d0 tx=0x7f6d4000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:06.783 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.779+0000 7f6d49ffb700 1 -- 172.21.15.22:0/1790327949 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d40009940 con 0x7f6d4c101c50 2024-03-24T09:46:06.783 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.779+0000 7f6d5311c700 1 -- 172.21.15.22:0/1790327949 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d4c196ae0 con 0x7f6d4c101c50 2024-03-24T09:46:06.783 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.779+0000 7f6d49ffb700 1 -- 172.21.15.22:0/1790327949 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6d40010460 con 0x7f6d4c101c50 2024-03-24T09:46:06.783 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.779+0000 7f6d49ffb700 1 -- 172.21.15.22:0/1790327949 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d4000f5f0 con 0x7f6d4c101c50 2024-03-24T09:46:06.784 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.780+0000 7f6d5311c700 1 -- 172.21.15.22:0/1790327949 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d4c197030 con 0x7f6d4c101c50 2024-03-24T09:46:06.784 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.780+0000 7f6d5311c700 1 -- 172.21.15.22:0/1790327949 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d4c105370 con 0x7f6d4c101c50 2024-03-24T09:46:06.784 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.781+0000 7f6d49ffb700 1 -- 172.21.15.22:0/1790327949 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f6d400105d0 con 0x7f6d4c101c50 2024-03-24T09:46:06.785 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.781+0000 7f6d49ffb700 1 --2- 172.21.15.22:0/1790327949 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f6d3c06cce0 0x7f6d3c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:06.785 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.781+0000 7f6d49ffb700 1 -- 172.21.15.22:0/1790327949 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f6d4008be40 con 0x7f6d4c101c50 2024-03-24T09:46:06.785 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.781+0000 7f6d4bfff700 1 --2- 172.21.15.22:0/1790327949 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f6d3c06cce0 0x7f6d3c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:06.785 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.782+0000 7f6d4bfff700 1 --2- 172.21.15.22:0/1790327949 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f6d3c06cce0 0x7f6d3c06f1a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6d3800b5c0 tx=0x7f6d38005e20 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:06.786 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.786+0000 7f6d49ffb700 1 -- 172.21.15.22:0/1790327949 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6d4005a060 con 0x7f6d4c101c50 2024-03-24T09:46:06.981 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.980+0000 7f6d5311c700 1 -- 172.21.15.22:0/1790327949 --> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6d4c1972a0 con 0x7f6d3c06cce0 2024-03-24T09:46:06.982 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.981+0000 7f6d49ffb700 1 -- 172.21.15.22:0/1790327949 <== mgr.14215 v2:172.21.15.22:6800/773863266 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27996 (secure 0 0 0) 0x7f6d4c1972a0 con 0x7f6d3c06cce0 2024-03-24T09:46:06.983 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:46:06.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.984+0000 7f6d5311c700 1 -- 172.21.15.22:0/1790327949 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f6d3c06cce0 msgr2=0x7f6d3c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:06.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.984+0000 7f6d5311c700 1 --2- 172.21.15.22:0/1790327949 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f6d3c06cce0 0x7f6d3c06f1a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6d3800b5c0 tx=0x7f6d38005e20 comp rx=0 tx=0).stop 2024-03-24T09:46:06.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.984+0000 7f6d5311c700 1 -- 172.21.15.22:0/1790327949 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6d4c101c50 msgr2=0x7f6d4c19c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:06.984 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.984+0000 7f6d5311c700 1 --2- 172.21.15.22:0/1790327949 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6d4c101c50 0x7f6d4c19c6c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6d4000d8d0 tx=0x7f6d4000dc90 comp rx=0 tx=0).stop 2024-03-24T09:46:06.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.984+0000 7f6d5311c700 1 -- 172.21.15.22:0/1790327949 shutdown_connections 2024-03-24T09:46:06.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.984+0000 7f6d5311c700 1 --2- 172.21.15.22:0/1790327949 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f6d3c06cce0 0x7f6d3c06f1a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:06.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.984+0000 7f6d5311c700 1 --2- 172.21.15.22:0/1790327949 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6d4c101c50 0x7f6d4c19c6c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:06.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.984+0000 7f6d5311c700 1 --2- 172.21.15.22:0/1790327949 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d4c102570 0x7f6d4c19cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:06.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.984+0000 7f6d5311c700 1 -- 172.21.15.22:0/1790327949 >> 172.21.15.22:0/1790327949 conn(0x7f6d4c076640 msgr2=0x7f6d4c102f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:06.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.984+0000 7f6d5311c700 1 -- 172.21.15.22:0/1790327949 shutdown_connections 2024-03-24T09:46:06.985 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:06.984+0000 7f6d5311c700 1 -- 172.21.15.22:0/1790327949 wait complete. 2024-03-24T09:46:06.985 INFO:teuthology.orchestra.run.smithi022.stderr:dumped all 2024-03-24T09:46:07.622 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:07 smithi022 ceph-mon[29512]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:46:07.646 INFO:teuthology.orchestra.run.smithi022.stdout:{"pg_ready":true,"pg_map":{"version":162,"stamp":"2024-03-24T09:46:06.498531+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":166008,"kb_used_data":3876,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":562304904,"statfs":{"total":575970213888,"available":575800221696,"internally_reserved":0,"allocated":3969024,"data_stored":2609166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":165927590},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001833"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":186,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-24T09:45:12.572979+0000","last_change":"2024-03-24T09:44:46.554917+0000","last_active":"2024-03-24T09:45:12.572979+0000","last_peered":"2024-03-24T09:45:12.572979+0000","last_clean":"2024-03-24T09:45:12.572979+0000","last_became_active":"2024-03-24T09:44:46.554634+0000","last_became_peered":"2024-03-24T09:44:46.554634+0000","last_unstale":"2024-03-24T09:45:12.572979+0000","last_undegraded":"2024-03-24T09:45:12.572979+0000","last_fullsized":"2024-03-24T09:45:12.572979+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:43:57.536831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:43:57.536831+0000","last_clean_scrub_stamp":"2024-03-24T09:43:57.536831+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:46:54.625430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724558,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27376,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717776,"statfs":{"total":95995035648,"available":95967002624,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]}]},{"osd":4,"up_from":34,"seq":146028888083,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27376,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717776,"statfs":{"total":95995035648,"available":95967002624,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 09:45:48 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.44600000000000001}]},{"osd":1,"last update":"Sun Mar 24 09:45:48 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.35399999999999998}]},{"osd":2,"last update":"Sun Mar 24 09:45:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.40300000000000002}]},{"osd":3,"last update":"Sun Mar 24 09:45:48 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.27900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 09:45:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.432},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.42399999999999999}]},{"osd":1,"last update":"Sun Mar 24 09:45:23 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.39300000000000002}]},{"osd":2,"last update":"Sun Mar 24 09:45:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.44600000000000001}]},{"osd":4,"last update":"Sun Mar 24 09:45:48 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.40500000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27376,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717776,"statfs":{"total":95995035648,"available":95967002624,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 09:46:01 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.34300000000000003,"15min":0.34699999999999998},"min":{"1min":0.16700000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.35799999999999998,"15min":0.36199999999999999},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.34699999999999998}]},{"osd":1,"last update":"Sun Mar 24 09:46:01 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.35099999999999998,"15min":0.35399999999999998},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.52100000000000002,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.378,"15min":0.38100000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"last":0.313}]},{"osd":3,"last update":"Sun Mar 24 09:45:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.47399999999999998}]},{"osd":4,"last update":"Sun Mar 24 09:45:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.48999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705704,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Mar 24 09:45:36 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.32300000000000001,"15min":0.32200000000000001},"min":{"1min":0.224,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.44700000000000001,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.33100000000000002,"15min":0.33000000000000002},"min":{"1min":0.23400000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.46300000000000002,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.20899999999999999}]},{"osd":2,"last update":"Sun Mar 24 09:46:00 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.36699999999999999,"15min":0.36899999999999999},"min":{"1min":0.24299999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.248},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.378,"15min":0.38},"min":{"1min":0.28499999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.26600000000000001}]},{"osd":3,"last update":"Sun Mar 24 09:45:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.44600000000000001}]},{"osd":4,"last update":"Sun Mar 24 09:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":6.141,"5min":6.141,"15min":6.141},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.39800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]}]},{"osd":1,"up_from":14,"seq":60129542179,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 09:45:33 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.33600000000000002,"15min":0.33000000000000002},"min":{"1min":0.255,"5min":0.216,"15min":0.216},"max":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.34100000000000003,"15min":0.33700000000000002},"min":{"1min":0.27900000000000003,"5min":0.219,"15min":0.219},"max":{"1min":0.50800000000000001,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.26200000000000001}]},{"osd":2,"last update":"Sun Mar 24 09:46:00 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.38,"15min":0.38100000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.52500000000000002,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.40200000000000002,"15min":0.40400000000000003},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.27300000000000002}]},{"osd":3,"last update":"Sun Mar 24 09:45:23 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.56100000000000005}]},{"osd":4,"last update":"Sun Mar 24 09:45:47 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.65000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T09:46:07.647 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-24T09:46:07.648 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-24T09:46:07.648 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-24T09:46:07.648 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph health --format=json 2024-03-24T09:46:07.861 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:46:07.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:07 smithi155 ceph-mon[37016]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:46:08.653 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:08 smithi022 ceph-mon[29512]: from='client.14440 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T09:46:08.653 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:08 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:46:08.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:08 smithi155 ceph-mon[37016]: from='client.14440 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T09:46:08.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:08 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:46:09.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.456+0000 7fa8728dd700 1 -- 172.21.15.22:0/847341596 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa86c0ff990 msgr2=0x7fa86c0ffe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:09.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.456+0000 7fa8728dd700 1 --2- 172.21.15.22:0/847341596 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa86c0ff990 0x7fa86c0ffe10 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa860009910 tx=0x7fa860009c20 comp rx=0 tx=0).stop 2024-03-24T09:46:09.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.456+0000 7fa8728dd700 1 -- 172.21.15.22:0/847341596 shutdown_connections 2024-03-24T09:46:09.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.456+0000 7fa8728dd700 1 --2- 172.21.15.22:0/847341596 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa86c0ff990 0x7fa86c0ffe10 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:09.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.456+0000 7fa8728dd700 1 --2- 172.21.15.22:0/847341596 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa86c105990 0x7fa86c105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:09.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.456+0000 7fa8728dd700 1 -- 172.21.15.22:0/847341596 >> 172.21.15.22:0/847341596 conn(0x7fa86c0fb4c0 msgr2=0x7fa86c0fd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:09.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.456+0000 7fa8728dd700 1 -- 172.21.15.22:0/847341596 shutdown_connections 2024-03-24T09:46:09.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.456+0000 7fa8728dd700 1 -- 172.21.15.22:0/847341596 wait complete. 2024-03-24T09:46:09.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.457+0000 7fa8728dd700 1 Processor -- start 2024-03-24T09:46:09.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.457+0000 7fa8728dd700 1 -- start start 2024-03-24T09:46:09.461 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.457+0000 7fa8728dd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa86c0ff990 0x7fa86c196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:09.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.457+0000 7fa8728dd700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa86c105990 0x7fa86c196f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:09.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.457+0000 7fa8728dd700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86c197550 con 0x7fa86c105990 2024-03-24T09:46:09.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.458+0000 7fa8728dd700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86c1976c0 con 0x7fa86c0ff990 2024-03-24T09:46:09.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.458+0000 7fa86b7fe700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa86c105990 0x7fa86c196f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:09.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.458+0000 7fa86b7fe700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa86c105990 0x7fa86c196f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:59458/0 (socket says 172.21.15.22:59458) 2024-03-24T09:46:09.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.458+0000 7fa86b7fe700 1 -- 172.21.15.22:0/1882928844 learned_addr learned my addr 172.21.15.22:0/1882928844 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:46:09.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.458+0000 7fa86bfff700 1 --2- 172.21.15.22:0/1882928844 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa86c0ff990 0x7fa86c196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:09.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.458+0000 7fa86b7fe700 1 -- 172.21.15.22:0/1882928844 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa86c0ff990 msgr2=0x7fa86c196a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:09.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.458+0000 7fa86b7fe700 1 --2- 172.21.15.22:0/1882928844 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa86c0ff990 0x7fa86c196a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:09.462 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.458+0000 7fa86b7fe700 1 -- 172.21.15.22:0/1882928844 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8600095f0 con 0x7fa86c105990 2024-03-24T09:46:09.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.458+0000 7fa86b7fe700 1 --2- 172.21.15.22:0/1882928844 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa86c105990 0x7fa86c196f50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa860006010 tx=0x7fa86000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:09.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.458+0000 7fa8697fa700 1 -- 172.21.15.22:0/1882928844 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa86001d070 con 0x7fa86c105990 2024-03-24T09:46:09.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.458+0000 7fa8728dd700 1 -- 172.21.15.22:0/1882928844 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa86c19b420 con 0x7fa86c105990 2024-03-24T09:46:09.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.458+0000 7fa8697fa700 1 -- 172.21.15.22:0/1882928844 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa86000f460 con 0x7fa86c105990 2024-03-24T09:46:09.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.458+0000 7fa8697fa700 1 -- 172.21.15.22:0/1882928844 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa860017630 con 0x7fa86c105990 2024-03-24T09:46:09.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.458+0000 7fa8728dd700 1 -- 172.21.15.22:0/1882928844 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa86c19b910 con 0x7fa86c105990 2024-03-24T09:46:09.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.460+0000 7fa8728dd700 1 -- 172.21.15.22:0/1882928844 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa86c1010d0 con 0x7fa86c105990 2024-03-24T09:46:09.463 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.460+0000 7fa8697fa700 1 -- 172.21.15.22:0/1882928844 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7fa86000f5d0 con 0x7fa86c105990 2024-03-24T09:46:09.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.460+0000 7fa8697fa700 1 --2- 172.21.15.22:0/1882928844 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fa85406cce0 0x7fa85406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:09.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.460+0000 7fa8697fa700 1 -- 172.21.15.22:0/1882928844 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fa86008d430 con 0x7fa86c105990 2024-03-24T09:46:09.464 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.460+0000 7fa86bfff700 1 --2- 172.21.15.22:0/1882928844 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fa85406cce0 0x7fa85406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:09.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.461+0000 7fa86bfff700 1 --2- 172.21.15.22:0/1882928844 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fa85406cce0 0x7fa85406f1a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa85c005ea0 tx=0x7fa85c005e30 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:09.465 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.465+0000 7fa8697fa700 1 -- 172.21.15.22:0/1882928844 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa86005b690 con 0x7fa86c105990 2024-03-24T09:46:09.717 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.717+0000 7fa8728dd700 1 -- 172.21.15.22:0/1882928844 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa86c1089a0 con 0x7fa86c105990 2024-03-24T09:46:09.718 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:09 smithi022 ceph-mon[29512]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:46:09.718 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.717+0000 7fa8697fa700 1 -- 172.21.15.22:0/1882928844 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fa86005b220 con 0x7fa86c105990 2024-03-24T09:46:09.719 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:46:09.719 INFO:teuthology.orchestra.run.smithi022.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-24T09:46:09.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.719+0000 7fa8728dd700 1 -- 172.21.15.22:0/1882928844 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fa85406cce0 msgr2=0x7fa85406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:09.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.719+0000 7fa8728dd700 1 --2- 172.21.15.22:0/1882928844 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fa85406cce0 0x7fa85406f1a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa85c005ea0 tx=0x7fa85c005e30 comp rx=0 tx=0).stop 2024-03-24T09:46:09.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.719+0000 7fa8728dd700 1 -- 172.21.15.22:0/1882928844 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa86c105990 msgr2=0x7fa86c196f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:09.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.719+0000 7fa8728dd700 1 --2- 172.21.15.22:0/1882928844 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa86c105990 0x7fa86c196f50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa860006010 tx=0x7fa86000be20 comp rx=0 tx=0).stop 2024-03-24T09:46:09.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.720+0000 7fa8728dd700 1 -- 172.21.15.22:0/1882928844 shutdown_connections 2024-03-24T09:46:09.720 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.720+0000 7fa8728dd700 1 --2- 172.21.15.22:0/1882928844 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fa85406cce0 0x7fa85406f1a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:09.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.720+0000 7fa8728dd700 1 --2- 172.21.15.22:0/1882928844 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa86c0ff990 0x7fa86c196a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:09.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.720+0000 7fa8728dd700 1 --2- 172.21.15.22:0/1882928844 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa86c105990 0x7fa86c196f50 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:09.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.720+0000 7fa8728dd700 1 -- 172.21.15.22:0/1882928844 >> 172.21.15.22:0/1882928844 conn(0x7fa86c0fb4c0 msgr2=0x7fa86c069530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:09.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.720+0000 7fa8728dd700 1 -- 172.21.15.22:0/1882928844 shutdown_connections 2024-03-24T09:46:09.721 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:09.720+0000 7fa8728dd700 1 -- 172.21.15.22:0/1882928844 wait complete. 2024-03-24T09:46:09.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:09 smithi155 ceph-mon[37016]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:46:10.389 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-24T09:46:10.390 INFO:tasks.cephadm:Setup complete, yielding 2024-03-24T09:46:10.390 INFO:teuthology.run_tasks:Running task print... 2024-03-24T09:46:10.399 INFO:teuthology.task.print:**** done end installing reef cephadm ... 2024-03-24T09:46:10.399 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T09:46:10.412 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi022.front.sepia.ceph.com 2024-03-24T09:46:10.413 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-03-24T09:46:10.623 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:46:10.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:10 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/1882928844' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T09:46:10.743 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:10 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/1882928844' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T09:46:11.879 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:11 smithi155 ceph-mon[37016]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:46:11.974 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:11 smithi022 ceph-mon[29512]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:46:12.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.244+0000 7fdc882b9700 1 -- 172.21.15.22:0/664504768 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdc800fedc0 msgr2=0x7fdc800ff1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:12.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.244+0000 7fdc882b9700 1 --2- 172.21.15.22:0/664504768 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdc800fedc0 0x7fdc800ff1e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fdc7000b0a0 tx=0x7fdc7000b3b0 comp rx=0 tx=0).stop 2024-03-24T09:46:12.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.244+0000 7fdc882b9700 1 -- 172.21.15.22:0/664504768 shutdown_connections 2024-03-24T09:46:12.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.244+0000 7fdc882b9700 1 --2- 172.21.15.22:0/664504768 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc80101e40 0x7fdc801022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:12.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.244+0000 7fdc882b9700 1 --2- 172.21.15.22:0/664504768 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdc800fedc0 0x7fdc800ff1e0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:12.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.244+0000 7fdc882b9700 1 -- 172.21.15.22:0/664504768 >> 172.21.15.22:0/664504768 conn(0x7fdc800fa700 msgr2=0x7fdc800fcb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:12.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.244+0000 7fdc882b9700 1 -- 172.21.15.22:0/664504768 shutdown_connections 2024-03-24T09:46:12.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.244+0000 7fdc882b9700 1 -- 172.21.15.22:0/664504768 wait complete. 2024-03-24T09:46:12.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.245+0000 7fdc882b9700 1 Processor -- start 2024-03-24T09:46:12.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.245+0000 7fdc882b9700 1 -- start start 2024-03-24T09:46:12.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.245+0000 7fdc882b9700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdc800fedc0 0x7fdc80196560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:12.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.245+0000 7fdc882b9700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc80101e40 0x7fdc80196aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:12.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.245+0000 7fdc882b9700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc801970c0 con 0x7fdc800fedc0 2024-03-24T09:46:12.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.245+0000 7fdc882b9700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc80197200 con 0x7fdc80101e40 2024-03-24T09:46:12.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.245+0000 7fdc86055700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdc800fedc0 0x7fdc80196560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:12.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.245+0000 7fdc86055700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdc800fedc0 0x7fdc80196560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:59480/0 (socket says 172.21.15.22:59480) 2024-03-24T09:46:12.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.245+0000 7fdc86055700 1 -- 172.21.15.22:0/2223802105 learned_addr learned my addr 172.21.15.22:0/2223802105 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:46:12.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.245+0000 7fdc85854700 1 --2- 172.21.15.22:0/2223802105 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc80101e40 0x7fdc80196aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:12.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.245+0000 7fdc86055700 1 -- 172.21.15.22:0/2223802105 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc80101e40 msgr2=0x7fdc80196aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:12.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.245+0000 7fdc86055700 1 --2- 172.21.15.22:0/2223802105 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc80101e40 0x7fdc80196aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:12.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.245+0000 7fdc86055700 1 -- 172.21.15.22:0/2223802105 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc70009d00 con 0x7fdc800fedc0 2024-03-24T09:46:12.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.245+0000 7fdc86055700 1 --2- 172.21.15.22:0/2223802105 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdc800fedc0 0x7fdc80196560 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fdc70006130 tx=0x7fdc7000cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:12.247 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.246+0000 7fdc7f7fe700 1 -- 172.21.15.22:0/2223802105 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc7000e040 con 0x7fdc800fedc0 2024-03-24T09:46:12.248 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.246+0000 7fdc882b9700 1 -- 172.21.15.22:0/2223802105 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc8019bc00 con 0x7fdc800fedc0 2024-03-24T09:46:12.248 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.246+0000 7fdc7f7fe700 1 -- 172.21.15.22:0/2223802105 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdc70003de0 con 0x7fdc800fedc0 2024-03-24T09:46:12.248 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.246+0000 7fdc7f7fe700 1 -- 172.21.15.22:0/2223802105 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc7001d850 con 0x7fdc800fedc0 2024-03-24T09:46:12.248 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.246+0000 7fdc882b9700 1 -- 172.21.15.22:0/2223802105 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc8019c0f0 con 0x7fdc800fedc0 2024-03-24T09:46:12.248 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.247+0000 7fdc882b9700 1 -- 172.21.15.22:0/2223802105 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc80190660 con 0x7fdc800fedc0 2024-03-24T09:46:12.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.247+0000 7fdc7f7fe700 1 -- 172.21.15.22:0/2223802105 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7fdc70004720 con 0x7fdc800fedc0 2024-03-24T09:46:12.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.248+0000 7fdc7f7fe700 1 --2- 172.21.15.22:0/2223802105 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fdc6806cce0 0x7fdc6806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:12.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.248+0000 7fdc7f7fe700 1 -- 172.21.15.22:0/2223802105 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fdc7008d720 con 0x7fdc800fedc0 2024-03-24T09:46:12.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.248+0000 7fdc85854700 1 --2- 172.21.15.22:0/2223802105 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fdc6806cce0 0x7fdc6806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:12.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.248+0000 7fdc85854700 1 --2- 172.21.15.22:0/2223802105 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fdc6806cce0 0x7fdc6806f1a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdc74005950 tx=0x7fdc7400b410 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:12.252 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.253+0000 7fdc7f7fe700 1 -- 172.21.15.22:0/2223802105 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fdc7005b8c0 con 0x7fdc800fedc0 2024-03-24T09:46:12.453 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.452+0000 7fdc882b9700 1 -- 172.21.15.22:0/2223802105 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7fdc8002cf60 con 0x7fdc800fedc0 2024-03-24T09:46:12.457 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.457+0000 7fdc7f7fe700 1 -- 172.21.15.22:0/2223802105 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7fdc7005b450 con 0x7fdc800fedc0 2024-03-24T09:46:12.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.459+0000 7fdc882b9700 1 -- 172.21.15.22:0/2223802105 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fdc6806cce0 msgr2=0x7fdc6806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:12.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.459+0000 7fdc882b9700 1 --2- 172.21.15.22:0/2223802105 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fdc6806cce0 0x7fdc6806f1a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdc74005950 tx=0x7fdc7400b410 comp rx=0 tx=0).stop 2024-03-24T09:46:12.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.459+0000 7fdc882b9700 1 -- 172.21.15.22:0/2223802105 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdc800fedc0 msgr2=0x7fdc80196560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:12.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.459+0000 7fdc882b9700 1 --2- 172.21.15.22:0/2223802105 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdc800fedc0 0x7fdc80196560 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fdc70006130 tx=0x7fdc7000cd20 comp rx=0 tx=0).stop 2024-03-24T09:46:12.459 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.459+0000 7fdc882b9700 1 -- 172.21.15.22:0/2223802105 shutdown_connections 2024-03-24T09:46:12.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.459+0000 7fdc882b9700 1 --2- 172.21.15.22:0/2223802105 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fdc6806cce0 0x7fdc6806f1a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:12.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.459+0000 7fdc882b9700 1 --2- 172.21.15.22:0/2223802105 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdc800fedc0 0x7fdc80196560 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:12.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.459+0000 7fdc882b9700 1 --2- 172.21.15.22:0/2223802105 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc80101e40 0x7fdc80196aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:12.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.459+0000 7fdc882b9700 1 -- 172.21.15.22:0/2223802105 >> 172.21.15.22:0/2223802105 conn(0x7fdc800fa700 msgr2=0x7fdc80105490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:12.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.459+0000 7fdc882b9700 1 -- 172.21.15.22:0/2223802105 shutdown_connections 2024-03-24T09:46:12.460 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:12.459+0000 7fdc882b9700 1 -- 172.21.15.22:0/2223802105 wait complete. 2024-03-24T09:46:13.144 INFO:teuthology.run_tasks:Running task print... 2024-03-24T09:46:13.157 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-03-24T09:46:13.157 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T09:46:13.166 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi022.front.sepia.ceph.com 2024-03-24T09:46:13.166 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- bash -c 'ceph orch status' 2024-03-24T09:46:13.383 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:46:13.717 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:13 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/2223802105' entity='client.admin' 2024-03-24T09:46:13.717 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:13 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:46:13.718 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:13 smithi022 ceph-mon[29512]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:46:13.718 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:13 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:46:13.718 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:13 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:46:13.718 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:13 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:13.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:13 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/2223802105' entity='client.admin' 2024-03-24T09:46:13.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:13 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:46:13.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:13 smithi155 ceph-mon[37016]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:46:13.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:13 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:46:13.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:13 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:46:13.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:13 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:14.926 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:14.925+0000 7f76a6416700 1 -- 172.21.15.22:0/994552683 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f76a0100c10 msgr2=0x7f76a0101070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:14.926 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:14.925+0000 7f76a6416700 1 --2- 172.21.15.22:0/994552683 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f76a0100c10 0x7f76a0101070 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7688009910 tx=0x7f7688009c20 comp rx=0 tx=0).stop 2024-03-24T09:46:14.926 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:14.925+0000 7f76a6416700 1 -- 172.21.15.22:0/994552683 shutdown_connections 2024-03-24T09:46:14.926 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:14.925+0000 7f76a6416700 1 --2- 172.21.15.22:0/994552683 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f76a0100c10 0x7f76a0101070 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:14.926 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:14.925+0000 7f76a6416700 1 --2- 172.21.15.22:0/994552683 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76a00ffa10 0x7f76a00ffe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:14.927 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:14.925+0000 7f76a6416700 1 -- 172.21.15.22:0/994552683 >> 172.21.15.22:0/994552683 conn(0x7f76a00faf70 msgr2=0x7f76a00fd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:14.927 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:14.925+0000 7f76a6416700 1 -- 172.21.15.22:0/994552683 shutdown_connections 2024-03-24T09:46:14.927 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:14.925+0000 7f76a6416700 1 -- 172.21.15.22:0/994552683 wait complete. 2024-03-24T09:46:14.927 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:14.926+0000 7f76a6416700 1 Processor -- start 2024-03-24T09:46:14.927 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:14.926+0000 7f76a6416700 1 -- start start 2024-03-24T09:46:14.929 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:14.926+0000 7f76a6416700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76a00ffa10 0x7f76a01964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:14.929 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:14.926+0000 7f76a6416700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f76a0100c10 0x7f76a0196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:14.929 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:14.926+0000 7f76a6416700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a0197030 con 0x7f76a0100c10 2024-03-24T09:46:14.930 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:14.926+0000 7f76a6416700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a0197170 con 0x7f76a00ffa10 2024-03-24T09:46:14.930 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:14.926+0000 7f769f7fe700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f76a0100c10 0x7f76a0196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:14.930 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:14.926+0000 7f769f7fe700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f76a0100c10 0x7f76a0196a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:59506/0 (socket says 172.21.15.22:59506) 2024-03-24T09:46:14.930 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:14.926+0000 7f769f7fe700 1 -- 172.21.15.22:0/4127670864 learned_addr learned my addr 172.21.15.22:0/4127670864 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:46:14.930 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:14.926+0000 7f769ffff700 1 --2- 172.21.15.22:0/4127670864 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76a00ffa10 0x7f76a01964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:14.930 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:14.927+0000 7f769f7fe700 1 -- 172.21.15.22:0/4127670864 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76a00ffa10 msgr2=0x7f76a01964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:14.930 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:14.927+0000 7f769f7fe700 1 --2- 172.21.15.22:0/4127670864 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76a00ffa10 0x7f76a01964d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:14.930 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:14.927+0000 7f769f7fe700 1 -- 172.21.15.22:0/4127670864 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76880095f0 con 0x7f76a0100c10 2024-03-24T09:46:14.930 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:14.927+0000 7f769f7fe700 1 --2- 172.21.15.22:0/4127670864 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f76a0100c10 0x7f76a0196a10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f7688006010 tx=0x7f768800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:14.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:14.927+0000 7f769d7fa700 1 -- 172.21.15.22:0/4127670864 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f768801d070 con 0x7f76a0100c10 2024-03-24T09:46:14.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:14.927+0000 7f76a6416700 1 -- 172.21.15.22:0/4127670864 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76a019bb70 con 0x7f76a0100c10 2024-03-24T09:46:14.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:14.927+0000 7f769d7fa700 1 -- 172.21.15.22:0/4127670864 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7688005dc0 con 0x7f76a0100c10 2024-03-24T09:46:14.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:14.927+0000 7f769d7fa700 1 -- 172.21.15.22:0/4127670864 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7688021640 con 0x7f76a0100c10 2024-03-24T09:46:14.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:14.927+0000 7f76a6416700 1 -- 172.21.15.22:0/4127670864 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76a019c0e0 con 0x7f76a0100c10 2024-03-24T09:46:14.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:14.928+0000 7f76a6416700 1 -- 172.21.15.22:0/4127670864 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76a0190690 con 0x7f76a0100c10 2024-03-24T09:46:14.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:14.928+0000 7f769d7fa700 1 -- 172.21.15.22:0/4127670864 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f768800f880 con 0x7f76a0100c10 2024-03-24T09:46:14.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:14.929+0000 7f769d7fa700 1 --2- 172.21.15.22:0/4127670864 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f768c06ccf0 0x7f768c06f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:14.931 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:14.929+0000 7f769d7fa700 1 -- 172.21.15.22:0/4127670864 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f768808d0c0 con 0x7f76a0100c10 2024-03-24T09:46:14.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:14.929+0000 7f769ffff700 1 --2- 172.21.15.22:0/4127670864 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f768c06ccf0 0x7f768c06f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:14.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:14.929+0000 7f769ffff700 1 --2- 172.21.15.22:0/4127670864 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f768c06ccf0 0x7f768c06f1b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7690005950 tx=0x7f76900058e0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:14.933 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:14.933+0000 7f769d7fa700 1 -- 172.21.15.22:0/4127670864 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f768805b2e0 con 0x7f76a0100c10 2024-03-24T09:46:15.152 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:15.152+0000 7f76a6416700 1 -- 172.21.15.22:0/4127670864 --> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f76a00611f0 con 0x7f768c06ccf0 2024-03-24T09:46:15.154 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:15.153+0000 7f769d7fa700 1 -- 172.21.15.22:0/4127670864 <== mgr.14215 v2:172.21.15.22:6800/773863266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f76a00611f0 con 0x7f768c06ccf0 2024-03-24T09:46:15.154 INFO:teuthology.orchestra.run.smithi022.stdout:Backend: cephadm 2024-03-24T09:46:15.154 INFO:teuthology.orchestra.run.smithi022.stdout:Available: Yes 2024-03-24T09:46:15.154 INFO:teuthology.orchestra.run.smithi022.stdout:Paused: No 2024-03-24T09:46:15.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:15.155+0000 7f76a6416700 1 -- 172.21.15.22:0/4127670864 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f768c06ccf0 msgr2=0x7f768c06f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:15.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:15.155+0000 7f76a6416700 1 --2- 172.21.15.22:0/4127670864 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f768c06ccf0 0x7f768c06f1b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7690005950 tx=0x7f76900058e0 comp rx=0 tx=0).stop 2024-03-24T09:46:15.156 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:15.155+0000 7f76a6416700 1 -- 172.21.15.22:0/4127670864 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f76a0100c10 msgr2=0x7f76a0196a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:15.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:15.155+0000 7f76a6416700 1 --2- 172.21.15.22:0/4127670864 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f76a0100c10 0x7f76a0196a10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f7688006010 tx=0x7f768800be20 comp rx=0 tx=0).stop 2024-03-24T09:46:15.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:15.156+0000 7f76a6416700 1 -- 172.21.15.22:0/4127670864 shutdown_connections 2024-03-24T09:46:15.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:15.156+0000 7f76a6416700 1 --2- 172.21.15.22:0/4127670864 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f768c06ccf0 0x7f768c06f1b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:15.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:15.156+0000 7f76a6416700 1 --2- 172.21.15.22:0/4127670864 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76a00ffa10 0x7f76a01964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:15.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:15.156+0000 7f76a6416700 1 --2- 172.21.15.22:0/4127670864 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f76a0100c10 0x7f76a0196a10 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:15.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:15.156+0000 7f76a6416700 1 -- 172.21.15.22:0/4127670864 >> 172.21.15.22:0/4127670864 conn(0x7f76a00faf70 msgr2=0x7f76a0104160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:15.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:15.156+0000 7f76a6416700 1 -- 172.21.15.22:0/4127670864 shutdown_connections 2024-03-24T09:46:15.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:15.156+0000 7f76a6416700 1 -- 172.21.15.22:0/4127670864 wait complete. 2024-03-24T09:46:15.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:15 smithi155 ceph-mon[37016]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:46:15.742 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:15 smithi022 ceph-mon[29512]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:46:15.766 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-24T09:46:15.979 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:46:16.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:16 smithi022 ceph-mon[29512]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:46:16.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:16 smithi155 ceph-mon[37016]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:46:17.669 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.667+0000 7fa47a169700 1 -- 172.21.15.22:0/57657783 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa474100c10 msgr2=0x7fa474101070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:17.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.667+0000 7fa47a169700 1 --2- 172.21.15.22:0/57657783 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa474100c10 0x7fa474101070 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa468009910 tx=0x7fa468009c20 comp rx=0 tx=0).stop 2024-03-24T09:46:17.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.667+0000 7fa47a169700 1 -- 172.21.15.22:0/57657783 shutdown_connections 2024-03-24T09:46:17.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.667+0000 7fa47a169700 1 --2- 172.21.15.22:0/57657783 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa474100c10 0x7fa474101070 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:17.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.667+0000 7fa47a169700 1 --2- 172.21.15.22:0/57657783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4740ffa10 0x7fa4740ffe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:17.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.667+0000 7fa47a169700 1 -- 172.21.15.22:0/57657783 >> 172.21.15.22:0/57657783 conn(0x7fa4740fafb0 msgr2=0x7fa4740fd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:17.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.667+0000 7fa47a169700 1 -- 172.21.15.22:0/57657783 shutdown_connections 2024-03-24T09:46:17.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.667+0000 7fa47a169700 1 -- 172.21.15.22:0/57657783 wait complete. 2024-03-24T09:46:17.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.667+0000 7fa47a169700 1 Processor -- start 2024-03-24T09:46:17.670 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.668+0000 7fa47a169700 1 -- start start 2024-03-24T09:46:17.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.668+0000 7fa47a169700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4740ffa10 0x7fa474069ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:17.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.668+0000 7fa47a169700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa474100c10 0x7fa47419b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:17.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.668+0000 7fa47a169700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa47419b840 con 0x7fa474100c10 2024-03-24T09:46:17.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.668+0000 7fa47a169700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa47419b980 con 0x7fa4740ffa10 2024-03-24T09:46:17.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.668+0000 7fa472ffd700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa474100c10 0x7fa47419b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:17.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.668+0000 7fa472ffd700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa474100c10 0x7fa47419b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:59524/0 (socket says 172.21.15.22:59524) 2024-03-24T09:46:17.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.668+0000 7fa472ffd700 1 -- 172.21.15.22:0/1885269714 learned_addr learned my addr 172.21.15.22:0/1885269714 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:46:17.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.668+0000 7fa4737fe700 1 --2- 172.21.15.22:0/1885269714 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4740ffa10 0x7fa474069ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:17.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.668+0000 7fa472ffd700 1 -- 172.21.15.22:0/1885269714 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4740ffa10 msgr2=0x7fa474069ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:17.671 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.668+0000 7fa472ffd700 1 --2- 172.21.15.22:0/1885269714 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4740ffa10 0x7fa474069ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:17.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.668+0000 7fa472ffd700 1 -- 172.21.15.22:0/1885269714 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4680095f0 con 0x7fa474100c10 2024-03-24T09:46:17.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.668+0000 7fa472ffd700 1 --2- 172.21.15.22:0/1885269714 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa474100c10 0x7fa47419b270 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa46800bd70 tx=0x7fa46800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:17.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.668+0000 7fa470ff9700 1 -- 172.21.15.22:0/1885269714 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa46801d070 con 0x7fa474100c10 2024-03-24T09:46:17.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.669+0000 7fa470ff9700 1 -- 172.21.15.22:0/1885269714 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa46800f460 con 0x7fa474100c10 2024-03-24T09:46:17.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.669+0000 7fa47a169700 1 -- 172.21.15.22:0/1885269714 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa47419bbb0 con 0x7fa474100c10 2024-03-24T09:46:17.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.669+0000 7fa470ff9700 1 -- 172.21.15.22:0/1885269714 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa468017630 con 0x7fa474100c10 2024-03-24T09:46:17.672 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.669+0000 7fa47a169700 1 -- 172.21.15.22:0/1885269714 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa47419c120 con 0x7fa474100c10 2024-03-24T09:46:17.673 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.670+0000 7fa47a169700 1 -- 172.21.15.22:0/1885269714 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa47418f590 con 0x7fa474100c10 2024-03-24T09:46:17.673 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.670+0000 7fa470ff9700 1 -- 172.21.15.22:0/1885269714 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7fa46800f5d0 con 0x7fa474100c10 2024-03-24T09:46:17.673 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.670+0000 7fa470ff9700 1 --2- 172.21.15.22:0/1885269714 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fa45c06cce0 0x7fa45c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:17.673 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.671+0000 7fa470ff9700 1 -- 172.21.15.22:0/1885269714 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fa46808d400 con 0x7fa474100c10 2024-03-24T09:46:17.674 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.671+0000 7fa4737fe700 1 --2- 172.21.15.22:0/1885269714 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fa45c06cce0 0x7fa45c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:17.674 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.671+0000 7fa4737fe700 1 --2- 172.21.15.22:0/1885269714 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fa45c06cce0 0x7fa45c06f1a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa464005fd0 tx=0x7fa464005dd0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:17.675 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.675+0000 7fa470ff9700 1 -- 172.21.15.22:0/1885269714 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa46805b620 con 0x7fa474100c10 2024-03-24T09:46:17.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:17 smithi022 ceph-mon[29512]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:46:17.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.887+0000 7fa47a169700 1 -- 172.21.15.22:0/1885269714 --> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4740611f0 con 0x7fa45c06cce0 2024-03-24T09:46:17.896 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.896+0000 7fa470ff9700 1 -- 172.21.15.22:0/1885269714 <== mgr.14215 v2:172.21.15.22:6800/773863266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3080 (secure 0 0 0) 0x7fa4740611f0 con 0x7fa45c06cce0 2024-03-24T09:46:17.896 INFO:teuthology.orchestra.run.smithi022.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T09:46:17.896 INFO:teuthology.orchestra.run.smithi022.stdout:alertmanager.smithi022 smithi022 *:9093,9094 running (3m) 2m ago 5m 26.0M - 0.25.0 c8568f914cd2 961b9fd08390 2024-03-24T09:46:17.896 INFO:teuthology.orchestra.run.smithi022.stdout:ceph-exporter.smithi022 smithi022 running (5m) 2m ago 5m 7046k - 18.2.2-952-g738c2fd6 68aff9aff3b2 513e822989e8 2024-03-24T09:46:17.896 INFO:teuthology.orchestra.run.smithi022.stdout:ceph-exporter.smithi155 smithi155 running (4m) 67s ago 4m 6983k - 18.2.2-952-g738c2fd6 68aff9aff3b2 2bc65c464d99 2024-03-24T09:46:17.897 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi022 smithi022 running (5m) 2m ago 5m 6665k - 18.2.2-952-g738c2fd6 68aff9aff3b2 a2a36122163c 2024-03-24T09:46:17.897 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi155 smithi155 running (4m) 67s ago 4m 6665k - 18.2.2-952-g738c2fd6 68aff9aff3b2 2d0b688c3a61 2024-03-24T09:46:17.897 INFO:teuthology.orchestra.run.smithi022.stdout:grafana.smithi022 smithi022 *:3000 running (3m) 2m ago 4m 78.3M - 9.4.7 954c08fa6188 082a79d8dba0 2024-03-24T09:46:17.897 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi022.emgmxz smithi022 *:9283,8765,8443 running (6m) 2m ago 6m 489M - 18.2.2-952-g738c2fd6 68aff9aff3b2 9c46d65fb5c7 2024-03-24T09:46:17.897 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi155.itwzew smithi155 *:8443,9283,8765 running (4m) 67s ago 4m 451M - 18.2.2-952-g738c2fd6 68aff9aff3b2 076d66e31d28 2024-03-24T09:46:17.897 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi022 smithi022 running (6m) 2m ago 6m 43.8M 2048M 18.2.2-952-g738c2fd6 68aff9aff3b2 59736938121b 2024-03-24T09:46:17.897 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi155 smithi155 running (4m) 67s ago 4m 40.1M 2048M 18.2.2-952-g738c2fd6 68aff9aff3b2 9188cfee0421 2024-03-24T09:46:17.897 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi022 smithi022 *:9100 running (5m) 2m ago 5m 15.4M - 1.5.0 0da6a335fe13 0734bfdc392d 2024-03-24T09:46:17.897 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi155 smithi155 *:9100 running (4m) 67s ago 4m 14.8M - 1.5.0 0da6a335fe13 18110dca979b 2024-03-24T09:46:17.897 INFO:teuthology.orchestra.run.smithi022.stdout:osd.0 smithi022 running (3m) 2m ago 3m 39.8M 3948M 18.2.2-952-g738c2fd6 68aff9aff3b2 db726da62a1c 2024-03-24T09:46:17.897 INFO:teuthology.orchestra.run.smithi022.stdout:osd.1 smithi022 running (2m) 2m ago 2m 41.0M 3948M 18.2.2-952-g738c2fd6 68aff9aff3b2 3c6af2f5b6f1 2024-03-24T09:46:17.898 INFO:teuthology.orchestra.run.smithi022.stdout:osd.2 smithi022 running (2m) 2m ago 2m 40.3M 3948M 18.2.2-952-g738c2fd6 68aff9aff3b2 99b3c8840e7d 2024-03-24T09:46:17.898 INFO:teuthology.orchestra.run.smithi022.stdout:osd.3 smithi155 running (2m) 67s ago 2m 40.9M 4970M 18.2.2-952-g738c2fd6 68aff9aff3b2 534b4e3f3f15 2024-03-24T09:46:17.898 INFO:teuthology.orchestra.run.smithi022.stdout:osd.4 smithi155 running (98s) 67s ago 98s 39.3M 4970M 18.2.2-952-g738c2fd6 68aff9aff3b2 a269f005acda 2024-03-24T09:46:17.898 INFO:teuthology.orchestra.run.smithi022.stdout:osd.5 smithi155 running (72s) 67s ago 72s 10.9M 4970M 18.2.2-952-g738c2fd6 68aff9aff3b2 20dd2513a0df 2024-03-24T09:46:17.898 INFO:teuthology.orchestra.run.smithi022.stdout:prometheus.smithi022 smithi022 *:9095 running (3m) 2m ago 4m 42.4M - 2.43.0 a07b618ecd1d 5cb914be4e25 2024-03-24T09:46:17.899 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.898+0000 7fa47a169700 1 -- 172.21.15.22:0/1885269714 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fa45c06cce0 msgr2=0x7fa45c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:17.899 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.898+0000 7fa47a169700 1 --2- 172.21.15.22:0/1885269714 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fa45c06cce0 0x7fa45c06f1a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa464005fd0 tx=0x7fa464005dd0 comp rx=0 tx=0).stop 2024-03-24T09:46:17.899 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.898+0000 7fa47a169700 1 -- 172.21.15.22:0/1885269714 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa474100c10 msgr2=0x7fa47419b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:17.899 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.898+0000 7fa47a169700 1 --2- 172.21.15.22:0/1885269714 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa474100c10 0x7fa47419b270 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa46800bd70 tx=0x7fa46800be50 comp rx=0 tx=0).stop 2024-03-24T09:46:17.899 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.898+0000 7fa47a169700 1 -- 172.21.15.22:0/1885269714 shutdown_connections 2024-03-24T09:46:17.899 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.898+0000 7fa47a169700 1 --2- 172.21.15.22:0/1885269714 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fa45c06cce0 0x7fa45c06f1a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:17.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.898+0000 7fa47a169700 1 --2- 172.21.15.22:0/1885269714 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4740ffa10 0x7fa474069ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:17.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.898+0000 7fa47a169700 1 --2- 172.21.15.22:0/1885269714 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa474100c10 0x7fa47419b270 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:17.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.898+0000 7fa47a169700 1 -- 172.21.15.22:0/1885269714 >> 172.21.15.22:0/1885269714 conn(0x7fa4740fafb0 msgr2=0x7fa474104210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:17.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.898+0000 7fa47a169700 1 -- 172.21.15.22:0/1885269714 shutdown_connections 2024-03-24T09:46:17.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:17.898+0000 7fa47a169700 1 -- 172.21.15.22:0/1885269714 wait complete. 2024-03-24T09:46:17.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:17 smithi155 ceph-mon[37016]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:46:18.690 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-24T09:46:18.903 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:46:19.855 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:19 smithi022 ceph-mon[29512]: from='client.14456 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:46:19.855 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:19 smithi022 ceph-mon[29512]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:46:19.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:19 smithi155 ceph-mon[37016]: from='client.14456 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:46:19.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:19 smithi155 ceph-mon[37016]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:46:20.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.349+0000 7f720ede3700 1 -- 172.21.15.22:0/456475682 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7208101ce0 msgr2=0x7f7208102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:20.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.349+0000 7f720ede3700 1 --2- 172.21.15.22:0/456475682 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7208101ce0 0x7f7208102140 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f71f4009910 tx=0x7f71f4009c20 comp rx=0 tx=0).stop 2024-03-24T09:46:20.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.349+0000 7f720ede3700 1 -- 172.21.15.22:0/456475682 shutdown_connections 2024-03-24T09:46:20.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.349+0000 7f720ede3700 1 --2- 172.21.15.22:0/456475682 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7208101ce0 0x7f7208102140 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:20.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.349+0000 7f720ede3700 1 --2- 172.21.15.22:0/456475682 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7208100ae0 0x7f7208100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:20.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.349+0000 7f720ede3700 1 -- 172.21.15.22:0/456475682 >> 172.21.15.22:0/456475682 conn(0x7f72080fc0a0 msgr2=0x7f72080fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:20.353 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.349+0000 7f720ede3700 1 -- 172.21.15.22:0/456475682 shutdown_connections 2024-03-24T09:46:20.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.349+0000 7f720ede3700 1 -- 172.21.15.22:0/456475682 wait complete. 2024-03-24T09:46:20.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.350+0000 7f720ede3700 1 Processor -- start 2024-03-24T09:46:20.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.350+0000 7f720ede3700 1 -- start start 2024-03-24T09:46:20.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.350+0000 7f720ede3700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7208100ae0 0x7f7208074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:20.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.350+0000 7f720ede3700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7208101ce0 0x7f72080731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:20.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.350+0000 7f720ede3700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72080736e0 con 0x7f7208100ae0 2024-03-24T09:46:20.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.350+0000 7f720ede3700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7208073820 con 0x7f7208101ce0 2024-03-24T09:46:20.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.350+0000 7f720cb7f700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7208100ae0 0x7f7208074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:20.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.350+0000 7f720cb7f700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7208100ae0 0x7f7208074b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:36212/0 (socket says 172.21.15.22:36212) 2024-03-24T09:46:20.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.350+0000 7f720cb7f700 1 -- 172.21.15.22:0/511478203 learned_addr learned my addr 172.21.15.22:0/511478203 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:46:20.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.351+0000 7f7207fff700 1 --2- 172.21.15.22:0/511478203 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7208101ce0 0x7f72080731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:20.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.351+0000 7f720cb7f700 1 -- 172.21.15.22:0/511478203 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7208101ce0 msgr2=0x7f72080731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:20.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.351+0000 7f720cb7f700 1 --2- 172.21.15.22:0/511478203 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7208101ce0 0x7f72080731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:20.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.351+0000 7f720cb7f700 1 -- 172.21.15.22:0/511478203 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f40095f0 con 0x7f7208100ae0 2024-03-24T09:46:20.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.351+0000 7f7207fff700 1 --2- 172.21.15.22:0/511478203 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7208101ce0 0x7f72080731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:46:20.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.351+0000 7f720cb7f700 1 --2- 172.21.15.22:0/511478203 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7208100ae0 0x7f7208074b00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f71fc00c8f0 tx=0x7f71fc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:20.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.351+0000 7f7205ffb700 1 -- 172.21.15.22:0/511478203 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f71fc007a70 con 0x7f7208100ae0 2024-03-24T09:46:20.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.351+0000 7f7205ffb700 1 -- 172.21.15.22:0/511478203 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f71fc007bd0 con 0x7f7208100ae0 2024-03-24T09:46:20.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.351+0000 7f720ede3700 1 -- 172.21.15.22:0/511478203 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7208073b00 con 0x7f7208100ae0 2024-03-24T09:46:20.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.351+0000 7f7205ffb700 1 -- 172.21.15.22:0/511478203 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f71fc018630 con 0x7f7208100ae0 2024-03-24T09:46:20.356 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.351+0000 7f720ede3700 1 -- 172.21.15.22:0/511478203 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f720819e520 con 0x7f7208100ae0 2024-03-24T09:46:20.356 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.352+0000 7f720ede3700 1 -- 172.21.15.22:0/511478203 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7208190660 con 0x7f7208100ae0 2024-03-24T09:46:20.356 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.353+0000 7f7205ffb700 1 -- 172.21.15.22:0/511478203 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f71fc00f450 con 0x7f7208100ae0 2024-03-24T09:46:20.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.353+0000 7f7205ffb700 1 --2- 172.21.15.22:0/511478203 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f71f806cce0 0x7f71f806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:20.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.353+0000 7f7205ffb700 1 -- 172.21.15.22:0/511478203 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f71fc08bf20 con 0x7f7208100ae0 2024-03-24T09:46:20.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.353+0000 7f7207fff700 1 --2- 172.21.15.22:0/511478203 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f71f806cce0 0x7f71f806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:20.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.353+0000 7f7207fff700 1 --2- 172.21.15.22:0/511478203 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f71f806cce0 0x7f71f806f1a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f71f4005b40 tx=0x7f71f4005ab0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:20.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.358+0000 7f7205ffb700 1 -- 172.21.15.22:0/511478203 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f71fc0bf9f0 con 0x7f7208100ae0 2024-03-24T09:46:20.566 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.566+0000 7f720ede3700 1 -- 172.21.15.22:0/511478203 --> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f72080611f0 con 0x7f71f806cce0 2024-03-24T09:46:20.571 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.570+0000 7f7205ffb700 1 -- 172.21.15.22:0/511478203 <== mgr.14215 v2:172.21.15.22:6800/773863266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1300 (secure 0 0 0) 0x7f72080611f0 con 0x7f71f806cce0 2024-03-24T09:46:20.571 INFO:teuthology.orchestra.run.smithi022.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-24T09:46:20.571 INFO:teuthology.orchestra.run.smithi022.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-03-24T09:46:20.571 INFO:teuthology.orchestra.run.smithi022.stdout:ceph-exporter 2/2 2m ago 6m * 2024-03-24T09:46:20.571 INFO:teuthology.orchestra.run.smithi022.stdout:crash 2/2 2m ago 6m * 2024-03-24T09:46:20.571 INFO:teuthology.orchestra.run.smithi022.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-03-24T09:46:20.572 INFO:teuthology.orchestra.run.smithi022.stdout:mgr 2/2 2m ago 6m count:2 2024-03-24T09:46:20.572 INFO:teuthology.orchestra.run.smithi022.stdout:mon 2/2 2m ago 5m smithi022:172.21.15.22=smithi022;smithi155:172.21.15.155=smithi155;count:2 2024-03-24T09:46:20.572 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-03-24T09:46:20.572 INFO:teuthology.orchestra.run.smithi022.stdout:osd 6 2m ago - 2024-03-24T09:46:20.572 INFO:teuthology.orchestra.run.smithi022.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-03-24T09:46:20.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.572+0000 7f720ede3700 1 -- 172.21.15.22:0/511478203 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f71f806cce0 msgr2=0x7f71f806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:20.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.573+0000 7f720ede3700 1 --2- 172.21.15.22:0/511478203 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f71f806cce0 0x7f71f806f1a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f71f4005b40 tx=0x7f71f4005ab0 comp rx=0 tx=0).stop 2024-03-24T09:46:20.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.573+0000 7f720ede3700 1 -- 172.21.15.22:0/511478203 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7208100ae0 msgr2=0x7f7208074b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:20.573 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.573+0000 7f720ede3700 1 --2- 172.21.15.22:0/511478203 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7208100ae0 0x7f7208074b00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f71fc00c8f0 tx=0x7f71fc00ccb0 comp rx=0 tx=0).stop 2024-03-24T09:46:20.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.573+0000 7f720ede3700 1 -- 172.21.15.22:0/511478203 shutdown_connections 2024-03-24T09:46:20.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.573+0000 7f720ede3700 1 --2- 172.21.15.22:0/511478203 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f71f806cce0 0x7f71f806f1a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:20.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.573+0000 7f720ede3700 1 --2- 172.21.15.22:0/511478203 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7208100ae0 0x7f7208074b00 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:20.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.573+0000 7f720ede3700 1 --2- 172.21.15.22:0/511478203 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7208101ce0 0x7f72080731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:20.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.573+0000 7f720ede3700 1 -- 172.21.15.22:0/511478203 >> 172.21.15.22:0/511478203 conn(0x7f72080fc0a0 msgr2=0x7f7208105280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:20.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.573+0000 7f720ede3700 1 -- 172.21.15.22:0/511478203 shutdown_connections 2024-03-24T09:46:20.574 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:20.573+0000 7f720ede3700 1 -- 172.21.15.22:0/511478203 wait complete. 2024-03-24T09:46:21.117 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-24T09:46:21.327 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:46:21.448 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:21 smithi022 ceph-mon[29512]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:46:21.448 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:21 smithi022 ceph-mon[29512]: from='client.14460 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:46:21.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:21 smithi155 ceph-mon[37016]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:46:21.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:21 smithi155 ceph-mon[37016]: from='client.14460 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:46:22.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:22.857+0000 7ffa88674700 1 -- 172.21.15.22:0/4050007784 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ffa80074de0 msgr2=0x7ffa80073240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:22.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:22.857+0000 7ffa88674700 1 --2- 172.21.15.22:0/4050007784 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ffa80074de0 0x7ffa80073240 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ffa78009910 tx=0x7ffa78009c20 comp rx=0 tx=0).stop 2024-03-24T09:46:22.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:22.857+0000 7ffa88674700 1 -- 172.21.15.22:0/4050007784 shutdown_connections 2024-03-24T09:46:22.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:22.857+0000 7ffa88674700 1 --2- 172.21.15.22:0/4050007784 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa80073810 0x7ffa80073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:22.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:22.857+0000 7ffa88674700 1 --2- 172.21.15.22:0/4050007784 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ffa80074de0 0x7ffa80073240 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:22.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:22.857+0000 7ffa88674700 1 -- 172.21.15.22:0/4050007784 >> 172.21.15.22:0/4050007784 conn(0x7ffa800fc090 msgr2=0x7ffa800fe4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:22.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:22.857+0000 7ffa88674700 1 -- 172.21.15.22:0/4050007784 shutdown_connections 2024-03-24T09:46:22.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:22.857+0000 7ffa88674700 1 -- 172.21.15.22:0/4050007784 wait complete. 2024-03-24T09:46:22.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:22.858+0000 7ffa88674700 1 Processor -- start 2024-03-24T09:46:22.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:22.858+0000 7ffa88674700 1 -- start start 2024-03-24T09:46:22.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:22.858+0000 7ffa88674700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ffa80073810 0x7ffa80198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:22.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:22.858+0000 7ffa88674700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa80074de0 0x7ffa80198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:22.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:22.858+0000 7ffa88674700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa80199220 con 0x7ffa80073810 2024-03-24T09:46:22.860 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:22.858+0000 7ffa88674700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa80199360 con 0x7ffa80074de0 2024-03-24T09:46:22.860 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:22.858+0000 7ffa86410700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ffa80073810 0x7ffa80198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:22.860 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:22.858+0000 7ffa86410700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ffa80073810 0x7ffa80198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:36222/0 (socket says 172.21.15.22:36222) 2024-03-24T09:46:22.860 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:22.858+0000 7ffa86410700 1 -- 172.21.15.22:0/1544087456 learned_addr learned my addr 172.21.15.22:0/1544087456 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:46:22.860 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:22.858+0000 7ffa85c0f700 1 --2- 172.21.15.22:0/1544087456 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa80074de0 0x7ffa80198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:22.860 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:22.858+0000 7ffa86410700 1 -- 172.21.15.22:0/1544087456 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa80074de0 msgr2=0x7ffa80198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:22.860 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:22.858+0000 7ffa86410700 1 --2- 172.21.15.22:0/1544087456 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa80074de0 0x7ffa80198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:22.860 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:22.858+0000 7ffa86410700 1 -- 172.21.15.22:0/1544087456 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa780095f0 con 0x7ffa80073810 2024-03-24T09:46:22.860 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:22.858+0000 7ffa86410700 1 --2- 172.21.15.22:0/1544087456 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ffa80073810 0x7ffa80198750 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ffa780059d0 tx=0x7ffa7800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:22.860 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:22.858+0000 7ffa7f7fe700 1 -- 172.21.15.22:0/1544087456 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa7801d070 con 0x7ffa80073810 2024-03-24T09:46:22.860 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:22.859+0000 7ffa7f7fe700 1 -- 172.21.15.22:0/1544087456 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffa7800f460 con 0x7ffa80073810 2024-03-24T09:46:22.860 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:22.859+0000 7ffa7f7fe700 1 -- 172.21.15.22:0/1544087456 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa78021640 con 0x7ffa80073810 2024-03-24T09:46:22.860 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:22.859+0000 7ffa88674700 1 -- 172.21.15.22:0/1544087456 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa8019dd20 con 0x7ffa80073810 2024-03-24T09:46:22.860 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:22.859+0000 7ffa88674700 1 -- 172.21.15.22:0/1544087456 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa8019e210 con 0x7ffa80073810 2024-03-24T09:46:22.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:22.859+0000 7ffa88674700 1 -- 172.21.15.22:0/1544087456 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa80192860 con 0x7ffa80073810 2024-03-24T09:46:22.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:22.860+0000 7ffa7f7fe700 1 -- 172.21.15.22:0/1544087456 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7ffa7800f5d0 con 0x7ffa80073810 2024-03-24T09:46:22.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:22.860+0000 7ffa7f7fe700 1 --2- 172.21.15.22:0/1544087456 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7ffa6806cce0 0x7ffa6806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:22.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:22.860+0000 7ffa7f7fe700 1 -- 172.21.15.22:0/1544087456 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ffa7808d180 con 0x7ffa80073810 2024-03-24T09:46:22.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:22.861+0000 7ffa85c0f700 1 --2- 172.21.15.22:0/1544087456 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7ffa6806cce0 0x7ffa6806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:22.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:22.861+0000 7ffa85c0f700 1 --2- 172.21.15.22:0/1544087456 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7ffa6806cce0 0x7ffa6806f1a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ffa80074b10 tx=0x7ffa7400a3b0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:22.865 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:22.865+0000 7ffa7f7fe700 1 -- 172.21.15.22:0/1544087456 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ffa7805b3a0 con 0x7ffa80073810 2024-03-24T09:46:23.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:23.073+0000 7ffa88674700 1 -- 172.21.15.22:0/1544087456 --> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7ffa800611f0 con 0x7ffa6806cce0 2024-03-24T09:46:23.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:23.074+0000 7ffa7f7fe700 1 -- 172.21.15.22:0/1544087456 <== mgr.14215 v2:172.21.15.22:6800/773863266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7ffa800611f0 con 0x7ffa6806cce0 2024-03-24T09:46:23.075 INFO:teuthology.orchestra.run.smithi022.stdout:HOST ADDR LABELS STATUS 2024-03-24T09:46:23.075 INFO:teuthology.orchestra.run.smithi022.stdout:smithi022 172.21.15.22 2024-03-24T09:46:23.075 INFO:teuthology.orchestra.run.smithi022.stdout:smithi155 172.21.15.155 2024-03-24T09:46:23.075 INFO:teuthology.orchestra.run.smithi022.stdout:2 hosts in cluster 2024-03-24T09:46:23.077 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:23.077+0000 7ffa88674700 1 -- 172.21.15.22:0/1544087456 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7ffa6806cce0 msgr2=0x7ffa6806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:23.077 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:23.077+0000 7ffa88674700 1 --2- 172.21.15.22:0/1544087456 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7ffa6806cce0 0x7ffa6806f1a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ffa80074b10 tx=0x7ffa7400a3b0 comp rx=0 tx=0).stop 2024-03-24T09:46:23.077 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:23.077+0000 7ffa88674700 1 -- 172.21.15.22:0/1544087456 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ffa80073810 msgr2=0x7ffa80198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:23.077 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:23.077+0000 7ffa88674700 1 --2- 172.21.15.22:0/1544087456 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ffa80073810 0x7ffa80198750 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ffa780059d0 tx=0x7ffa7800bd70 comp rx=0 tx=0).stop 2024-03-24T09:46:23.077 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:23.077+0000 7ffa88674700 1 -- 172.21.15.22:0/1544087456 shutdown_connections 2024-03-24T09:46:23.078 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:23.077+0000 7ffa88674700 1 --2- 172.21.15.22:0/1544087456 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7ffa6806cce0 0x7ffa6806f1a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:23.078 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:23.077+0000 7ffa88674700 1 --2- 172.21.15.22:0/1544087456 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ffa80073810 0x7ffa80198750 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:23.078 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:23.077+0000 7ffa88674700 1 --2- 172.21.15.22:0/1544087456 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa80074de0 0x7ffa80198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:23.078 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:23.077+0000 7ffa88674700 1 -- 172.21.15.22:0/1544087456 >> 172.21.15.22:0/1544087456 conn(0x7ffa800fc090 msgr2=0x7ffa80106cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:23.078 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:23.077+0000 7ffa88674700 1 -- 172.21.15.22:0/1544087456 shutdown_connections 2024-03-24T09:46:23.078 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:23.077+0000 7ffa88674700 1 -- 172.21.15.22:0/1544087456 wait complete. 2024-03-24T09:46:23.731 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:23 smithi022 ceph-mon[29512]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:46:23.731 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:23 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:46:23.754 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-24T09:46:23.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:23 smithi155 ceph-mon[37016]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:46:23.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:23 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:46:23.963 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:46:24.736 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:24 smithi022 ceph-mon[29512]: from='client.14464 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:46:24.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:24 smithi155 ceph-mon[37016]: from='client.14464 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:46:25.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.492+0000 7f9eb1ef9700 1 -- 172.21.15.22:0/39690473 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9eac101ce0 msgr2=0x7f9eac102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:25.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.492+0000 7f9eb1ef9700 1 --2- 172.21.15.22:0/39690473 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9eac101ce0 0x7f9eac102140 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9e94009910 tx=0x7f9e94009c20 comp rx=0 tx=0).stop 2024-03-24T09:46:25.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.492+0000 7f9eb1ef9700 1 -- 172.21.15.22:0/39690473 shutdown_connections 2024-03-24T09:46:25.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.492+0000 7f9eb1ef9700 1 --2- 172.21.15.22:0/39690473 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9eac101ce0 0x7f9eac102140 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:25.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.492+0000 7f9eb1ef9700 1 --2- 172.21.15.22:0/39690473 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9eac100ae0 0x7f9eac100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:25.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.492+0000 7f9eb1ef9700 1 -- 172.21.15.22:0/39690473 >> 172.21.15.22:0/39690473 conn(0x7f9eac0fc0a0 msgr2=0x7f9eac0fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:25.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.492+0000 7f9eb1ef9700 1 -- 172.21.15.22:0/39690473 shutdown_connections 2024-03-24T09:46:25.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.492+0000 7f9eb1ef9700 1 -- 172.21.15.22:0/39690473 wait complete. 2024-03-24T09:46:25.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.492+0000 7f9eb1ef9700 1 Processor -- start 2024-03-24T09:46:25.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.492+0000 7f9eb1ef9700 1 -- start start 2024-03-24T09:46:25.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.493+0000 7f9eb1ef9700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9eac100ae0 0x7f9eac1964c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:25.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.493+0000 7f9eb1ef9700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9eac101ce0 0x7f9eac196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:25.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.493+0000 7f9eb1ef9700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eac197020 con 0x7f9eac100ae0 2024-03-24T09:46:25.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.493+0000 7f9eb1ef9700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eac197160 con 0x7f9eac101ce0 2024-03-24T09:46:25.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.493+0000 7f9eab7fe700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9eac100ae0 0x7f9eac1964c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:25.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.493+0000 7f9eab7fe700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9eac100ae0 0x7f9eac1964c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:36236/0 (socket says 172.21.15.22:36236) 2024-03-24T09:46:25.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.493+0000 7f9eab7fe700 1 -- 172.21.15.22:0/1753139253 learned_addr learned my addr 172.21.15.22:0/1753139253 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:46:25.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.493+0000 7f9ea3fff700 1 --2- 172.21.15.22:0/1753139253 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9eac101ce0 0x7f9eac196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:25.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.493+0000 7f9eab7fe700 1 -- 172.21.15.22:0/1753139253 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9eac101ce0 msgr2=0x7f9eac196a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:25.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.493+0000 7f9eab7fe700 1 --2- 172.21.15.22:0/1753139253 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9eac101ce0 0x7f9eac196a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:25.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.493+0000 7f9eab7fe700 1 -- 172.21.15.22:0/1753139253 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e940095f0 con 0x7f9eac100ae0 2024-03-24T09:46:25.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.493+0000 7f9eab7fe700 1 --2- 172.21.15.22:0/1753139253 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9eac100ae0 0x7f9eac1964c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c00d900 tx=0x7f9e9c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:25.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.493+0000 7f9ea97fa700 1 -- 172.21.15.22:0/1753139253 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e9c0041f0 con 0x7f9eac100ae0 2024-03-24T09:46:25.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.493+0000 7f9ea97fa700 1 -- 172.21.15.22:0/1753139253 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9e9c004350 con 0x7f9eac100ae0 2024-03-24T09:46:25.497 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.493+0000 7f9eb1ef9700 1 -- 172.21.15.22:0/1753139253 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eac19bbc0 con 0x7f9eac100ae0 2024-03-24T09:46:25.498 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.493+0000 7f9ea97fa700 1 -- 172.21.15.22:0/1753139253 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e9c003cf0 con 0x7f9eac100ae0 2024-03-24T09:46:25.498 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.493+0000 7f9eb1ef9700 1 -- 172.21.15.22:0/1753139253 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eac19c110 con 0x7f9eac100ae0 2024-03-24T09:46:25.498 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.494+0000 7f9eb1ef9700 1 -- 172.21.15.22:0/1753139253 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9eac190660 con 0x7f9eac100ae0 2024-03-24T09:46:25.498 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.495+0000 7f9ea97fa700 1 -- 172.21.15.22:0/1753139253 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f9e9c009730 con 0x7f9eac100ae0 2024-03-24T09:46:25.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.495+0000 7f9ea97fa700 1 --2- 172.21.15.22:0/1753139253 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f9e9806cce0 0x7f9e9806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:25.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.495+0000 7f9ea97fa700 1 -- 172.21.15.22:0/1753139253 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f9e9c08bd70 con 0x7f9eac100ae0 2024-03-24T09:46:25.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.495+0000 7f9ea3fff700 1 --2- 172.21.15.22:0/1753139253 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f9e9806cce0 0x7f9e9806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:25.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.496+0000 7f9ea3fff700 1 --2- 172.21.15.22:0/1753139253 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f9e9806cce0 0x7f9e9806f1a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9e94005090 tx=0x7f9e94005960 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:25.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.500+0000 7f9ea97fa700 1 -- 172.21.15.22:0/1753139253 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9e9c059f10 con 0x7f9eac100ae0 2024-03-24T09:46:25.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:25 smithi155 ceph-mon[37016]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:46:25.712 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.711+0000 7f9eb1ef9700 1 -- 172.21.15.22:0/1753139253 --> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f9eac0611f0 con 0x7f9e9806cce0 2024-03-24T09:46:25.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.713+0000 7f9ea97fa700 1 -- 172.21.15.22:0/1753139253 <== mgr.14215 v2:172.21.15.22:6800/773863266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7f9eac0611f0 con 0x7f9e9806cce0 2024-03-24T09:46:25.713 INFO:teuthology.orchestra.run.smithi022.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-24T09:46:25.714 INFO:teuthology.orchestra.run.smithi022.stdout:smithi022 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB747402EC480DGN 447G No 2m ago Has a FileSystem, LVM detected 2024-03-24T09:46:25.714 INFO:teuthology.orchestra.run.smithi022.stdout:smithi155 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300GW400BGN 372G No 71s ago Has a FileSystem, LVM detected 2024-03-24T09:46:25.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.715+0000 7f9eb1ef9700 1 -- 172.21.15.22:0/1753139253 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f9e9806cce0 msgr2=0x7f9e9806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:25.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.715+0000 7f9eb1ef9700 1 --2- 172.21.15.22:0/1753139253 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f9e9806cce0 0x7f9e9806f1a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9e94005090 tx=0x7f9e94005960 comp rx=0 tx=0).stop 2024-03-24T09:46:25.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.715+0000 7f9eb1ef9700 1 -- 172.21.15.22:0/1753139253 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9eac100ae0 msgr2=0x7f9eac1964c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:25.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.715+0000 7f9eb1ef9700 1 --2- 172.21.15.22:0/1753139253 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9eac100ae0 0x7f9eac1964c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c00d900 tx=0x7f9e9c00dc10 comp rx=0 tx=0).stop 2024-03-24T09:46:25.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.715+0000 7f9eb1ef9700 1 -- 172.21.15.22:0/1753139253 shutdown_connections 2024-03-24T09:46:25.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.715+0000 7f9eb1ef9700 1 --2- 172.21.15.22:0/1753139253 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f9e9806cce0 0x7f9e9806f1a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:25.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.715+0000 7f9eb1ef9700 1 --2- 172.21.15.22:0/1753139253 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9eac100ae0 0x7f9eac1964c0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:25.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.715+0000 7f9eb1ef9700 1 --2- 172.21.15.22:0/1753139253 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9eac101ce0 0x7f9eac196a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:25.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.715+0000 7f9eb1ef9700 1 -- 172.21.15.22:0/1753139253 >> 172.21.15.22:0/1753139253 conn(0x7f9eac0fc0a0 msgr2=0x7f9eac105280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:25.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.715+0000 7f9eb1ef9700 1 -- 172.21.15.22:0/1753139253 shutdown_connections 2024-03-24T09:46:25.716 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:25.715+0000 7f9eb1ef9700 1 -- 172.21.15.22:0/1753139253 wait complete. 2024-03-24T09:46:26.009 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:25 smithi022 ceph-mon[29512]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:46:26.306 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T09:46:26.317 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi022.front.sepia.ceph.com 2024-03-24T09:46:26.318 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-03-24T09:46:26.528 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:46:27.901 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:27 smithi022 ceph-mon[29512]: from='client.14468 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:46:27.901 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:27 smithi022 ceph-mon[29512]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:46:27.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:27 smithi155 ceph-mon[37016]: from='client.14468 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:46:27.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:27 smithi155 ceph-mon[37016]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:46:28.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:28.116+0000 7f706da70700 1 -- 172.21.15.22:0/302974869 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7068101ce0 msgr2=0x7f7068102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:28.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:28.116+0000 7f706da70700 1 --2- 172.21.15.22:0/302974869 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7068101ce0 0x7f7068102140 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7050009910 tx=0x7f7050009c20 comp rx=0 tx=0).stop 2024-03-24T09:46:28.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:28.116+0000 7f706da70700 1 -- 172.21.15.22:0/302974869 shutdown_connections 2024-03-24T09:46:28.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:28.116+0000 7f706da70700 1 --2- 172.21.15.22:0/302974869 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7068101ce0 0x7f7068102140 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:28.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:28.116+0000 7f706da70700 1 --2- 172.21.15.22:0/302974869 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7068100ae0 0x7f7068100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:28.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:28.116+0000 7f706da70700 1 -- 172.21.15.22:0/302974869 >> 172.21.15.22:0/302974869 conn(0x7f70680fc0a0 msgr2=0x7f70680fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:28.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:28.116+0000 7f706da70700 1 -- 172.21.15.22:0/302974869 shutdown_connections 2024-03-24T09:46:28.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:28.116+0000 7f706da70700 1 -- 172.21.15.22:0/302974869 wait complete. 2024-03-24T09:46:28.118 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:28.116+0000 7f706da70700 1 Processor -- start 2024-03-24T09:46:28.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:28.116+0000 7f706da70700 1 -- start start 2024-03-24T09:46:28.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:28.117+0000 7f706da70700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7068100ae0 0x7f7068074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:28.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:28.117+0000 7f706da70700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7068101ce0 0x7f70680731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:28.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:28.117+0000 7f706da70700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70680736e0 con 0x7f7068101ce0 2024-03-24T09:46:28.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:28.117+0000 7f706da70700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7068073820 con 0x7f7068100ae0 2024-03-24T09:46:28.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:28.117+0000 7f7066ffd700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7068101ce0 0x7f70680731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:28.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:28.117+0000 7f7066ffd700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7068101ce0 0x7f70680731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:36248/0 (socket says 172.21.15.22:36248) 2024-03-24T09:46:28.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:28.117+0000 7f7066ffd700 1 -- 172.21.15.22:0/4239371208 learned_addr learned my addr 172.21.15.22:0/4239371208 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:46:28.119 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:28.117+0000 7f70677fe700 1 --2- 172.21.15.22:0/4239371208 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7068100ae0 0x7f7068074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:28.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:28.117+0000 7f7066ffd700 1 -- 172.21.15.22:0/4239371208 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7068100ae0 msgr2=0x7f7068074b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:28.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:28.117+0000 7f7066ffd700 1 --2- 172.21.15.22:0/4239371208 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7068100ae0 0x7f7068074b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:28.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:28.117+0000 7f7066ffd700 1 -- 172.21.15.22:0/4239371208 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70500095f0 con 0x7f7068101ce0 2024-03-24T09:46:28.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:28.117+0000 7f7066ffd700 1 --2- 172.21.15.22:0/4239371208 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7068101ce0 0x7f70680731a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f70500092e0 tx=0x7f705000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:28.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:28.117+0000 7f7064ff9700 1 -- 172.21.15.22:0/4239371208 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f705001d070 con 0x7f7068101ce0 2024-03-24T09:46:28.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:28.117+0000 7f7064ff9700 1 -- 172.21.15.22:0/4239371208 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f705000f460 con 0x7f7068101ce0 2024-03-24T09:46:28.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:28.117+0000 7f706da70700 1 -- 172.21.15.22:0/4239371208 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7068073aa0 con 0x7f7068101ce0 2024-03-24T09:46:28.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:28.117+0000 7f7064ff9700 1 -- 172.21.15.22:0/4239371208 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7050021620 con 0x7f7068101ce0 2024-03-24T09:46:28.120 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:28.117+0000 7f706da70700 1 -- 172.21.15.22:0/4239371208 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7068074010 con 0x7f7068101ce0 2024-03-24T09:46:28.121 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:28.119+0000 7f706da70700 1 -- 172.21.15.22:0/4239371208 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7068190660 con 0x7f7068101ce0 2024-03-24T09:46:28.121 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:28.119+0000 7f7064ff9700 1 -- 172.21.15.22:0/4239371208 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f705000f5d0 con 0x7f7068101ce0 2024-03-24T09:46:28.122 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:28.119+0000 7f7064ff9700 1 --2- 172.21.15.22:0/4239371208 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f705406cce0 0x7f705406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:28.122 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:28.119+0000 7f7064ff9700 1 -- 172.21.15.22:0/4239371208 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f705008cef0 con 0x7f7068101ce0 2024-03-24T09:46:28.122 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:28.119+0000 7f70677fe700 1 --2- 172.21.15.22:0/4239371208 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f705406cce0 0x7f705406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:28.122 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:28.120+0000 7f70677fe700 1 --2- 172.21.15.22:0/4239371208 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f705406cce0 0x7f705406f1a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7058009550 tx=0x7f70580091b0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:28.124 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:28.124+0000 7f7064ff9700 1 -- 172.21.15.22:0/4239371208 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f705005b110 con 0x7f7068101ce0 2024-03-24T09:46:28.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:28.354+0000 7f706da70700 1 -- 172.21.15.22:0/4239371208 --> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f7068074010 con 0x7f705406cce0 2024-03-24T09:46:29.211 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:28 smithi155 ceph-mon[37016]: from='client.14472 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:46:29.211 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:28 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-03-24T09:46:29.211 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:28 smithi155 ceph-mon[37016]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:46:29.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:28 smithi022 ceph-mon[29512]: from='client.14472 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:46:29.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:28 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-03-24T09:46:29.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:28 smithi022 ceph-mon[29512]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:46:29.685 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:29.684+0000 7f7064ff9700 1 -- 172.21.15.22:0/4239371208 <== mgr.14215 v2:172.21.15.22:6800/773863266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7068074010 con 0x7f705406cce0 2024-03-24T09:46:29.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:29.688+0000 7f706da70700 1 -- 172.21.15.22:0/4239371208 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f705406cce0 msgr2=0x7f705406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:29.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:29.688+0000 7f706da70700 1 --2- 172.21.15.22:0/4239371208 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f705406cce0 0x7f705406f1a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7058009550 tx=0x7f70580091b0 comp rx=0 tx=0).stop 2024-03-24T09:46:29.689 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:29.688+0000 7f706da70700 1 -- 172.21.15.22:0/4239371208 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7068101ce0 msgr2=0x7f70680731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:29.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:29.688+0000 7f706da70700 1 --2- 172.21.15.22:0/4239371208 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7068101ce0 0x7f70680731a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f70500092e0 tx=0x7f705000bd70 comp rx=0 tx=0).stop 2024-03-24T09:46:29.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:29.688+0000 7f706da70700 1 -- 172.21.15.22:0/4239371208 shutdown_connections 2024-03-24T09:46:29.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:29.688+0000 7f706da70700 1 --2- 172.21.15.22:0/4239371208 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f705406cce0 0x7f705406f1a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:29.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:29.688+0000 7f706da70700 1 --2- 172.21.15.22:0/4239371208 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7068100ae0 0x7f7068074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:29.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:29.688+0000 7f706da70700 1 --2- 172.21.15.22:0/4239371208 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7068101ce0 0x7f70680731a0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:29.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:29.688+0000 7f706da70700 1 -- 172.21.15.22:0/4239371208 >> 172.21.15.22:0/4239371208 conn(0x7f70680fc0a0 msgr2=0x7f7068105280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:29.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:29.689+0000 7f706da70700 1 -- 172.21.15.22:0/4239371208 shutdown_connections 2024-03-24T09:46:29.690 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:29.689+0000 7f706da70700 1 -- 172.21.15.22:0/4239371208 wait complete. 2024-03-24T09:46:29.908 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:46:29.664+0000 7f6a466e0700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T09:46:30.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:29 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-03-24T09:46:30.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:29 smithi155 ceph-mon[37016]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T09:46:30.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:29 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-03-24T09:46:30.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:29 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-03-24T09:46:30.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:29 smithi155 ceph-mon[37016]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T09:46:30.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:29 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-03-24T09:46:30.213 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:29 smithi155 ceph-mon[37016]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T09:46:30.213 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:29 smithi155 ceph-mon[37016]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-24T09:46:30.213 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:29 smithi155 ceph-mon[37016]: osdmap e43: 6 total, 6 up, 6 in 2024-03-24T09:46:30.213 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:29 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-03-24T09:46:30.213 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:29 smithi155 ceph-mon[37016]: fsmap cephfs:0 2024-03-24T09:46:30.213 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:29 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:30.213 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:29 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:46:30.213 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:29 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:46:30.213 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:29 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:46:30.213 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:29 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:30.213 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:29 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi022.mhcbrm", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:46:30.214 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:29 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi022.mhcbrm", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:46:30.214 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:29 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:46:30.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:29 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-03-24T09:46:30.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:29 smithi022 ceph-mon[29512]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T09:46:30.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:29 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-03-24T09:46:30.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:29 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-03-24T09:46:30.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:29 smithi022 ceph-mon[29512]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T09:46:30.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:29 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-03-24T09:46:30.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:29 smithi022 ceph-mon[29512]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T09:46:30.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:29 smithi022 ceph-mon[29512]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-24T09:46:30.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:29 smithi022 ceph-mon[29512]: osdmap e43: 6 total, 6 up, 6 in 2024-03-24T09:46:30.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:29 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-03-24T09:46:30.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:29 smithi022 ceph-mon[29512]: fsmap cephfs:0 2024-03-24T09:46:30.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:29 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:30.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:29 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:46:30.284 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:29 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:46:30.284 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:29 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:46:30.284 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:29 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:30.284 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:29 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi022.mhcbrm", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:46:30.284 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:29 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi022.mhcbrm", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:46:30.284 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:29 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:46:30.553 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-24T09:46:31.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:31 smithi155 ceph-mon[37016]: Saving service mds.cephfs spec with placement count:4 2024-03-24T09:46:31.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:31 smithi155 ceph-mon[37016]: Deploying daemon mds.cephfs.smithi022.mhcbrm on smithi022 2024-03-24T09:46:31.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:31 smithi155 ceph-mon[37016]: pgmap v177: 65 pgs: 3 creating+peering, 1 active+clean, 61 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:46:31.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:31 smithi155 ceph-mon[37016]: osdmap e44: 6 total, 6 up, 6 in 2024-03-24T09:46:31.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:31 smithi022 ceph-mon[29512]: Saving service mds.cephfs spec with placement count:4 2024-03-24T09:46:31.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:31 smithi022 ceph-mon[29512]: Deploying daemon mds.cephfs.smithi022.mhcbrm on smithi022 2024-03-24T09:46:31.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:31 smithi022 ceph-mon[29512]: pgmap v177: 65 pgs: 3 creating+peering, 1 active+clean, 61 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:46:31.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:31 smithi022 ceph-mon[29512]: osdmap e44: 6 total, 6 up, 6 in 2024-03-24T09:46:31.651 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:46:32.944 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:32 smithi022 ceph-mon[29512]: osdmap e45: 6 total, 6 up, 6 in 2024-03-24T09:46:32.944 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:32 smithi022 ceph-mon[29512]: pgmap v180: 65 pgs: 10 active+clean, 9 creating+peering, 46 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:46:32.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:32 smithi155 ceph-mon[37016]: osdmap e45: 6 total, 6 up, 6 in 2024-03-24T09:46:32.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:32 smithi155 ceph-mon[37016]: pgmap v180: 65 pgs: 10 active+clean, 9 creating+peering, 46 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:46:33.471 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.468+0000 7f0ec23e6700 1 -- 172.21.15.22:0/3303314483 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0ebc100bd0 msgr2=0x7f0ebc101030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:33.471 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.468+0000 7f0ec23e6700 1 --2- 172.21.15.22:0/3303314483 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0ebc100bd0 0x7f0ebc101030 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0eb0009910 tx=0x7f0eb0009c20 comp rx=0 tx=0).stop 2024-03-24T09:46:33.471 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.468+0000 7f0ec23e6700 1 -- 172.21.15.22:0/3303314483 shutdown_connections 2024-03-24T09:46:33.471 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.468+0000 7f0ec23e6700 1 --2- 172.21.15.22:0/3303314483 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0ebc100bd0 0x7f0ebc101030 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:33.471 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.468+0000 7f0ec23e6700 1 --2- 172.21.15.22:0/3303314483 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0ebc0ff9d0 0x7f0ebc0ffdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:33.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.468+0000 7f0ec23e6700 1 -- 172.21.15.22:0/3303314483 >> 172.21.15.22:0/3303314483 conn(0x7f0ebc0faf70 msgr2=0x7f0ebc0fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:33.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.468+0000 7f0ec23e6700 1 -- 172.21.15.22:0/3303314483 shutdown_connections 2024-03-24T09:46:33.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.468+0000 7f0ec23e6700 1 -- 172.21.15.22:0/3303314483 wait complete. 2024-03-24T09:46:33.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.469+0000 7f0ec23e6700 1 Processor -- start 2024-03-24T09:46:33.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.469+0000 7f0ec23e6700 1 -- start start 2024-03-24T09:46:33.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.469+0000 7f0ec23e6700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0ebc0ff9d0 0x7f0ebc1966a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:33.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.469+0000 7f0ec23e6700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0ebc100bd0 0x7f0ebc196be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:33.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.469+0000 7f0ec23e6700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ebc197200 con 0x7f0ebc100bd0 2024-03-24T09:46:33.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.469+0000 7f0ec23e6700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ebc197340 con 0x7f0ebc0ff9d0 2024-03-24T09:46:33.472 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.469+0000 7f0ebb7fe700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0ebc100bd0 0x7f0ebc196be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:33.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.469+0000 7f0ebb7fe700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0ebc100bd0 0x7f0ebc196be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:51490/0 (socket says 172.21.15.22:51490) 2024-03-24T09:46:33.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.469+0000 7f0ebb7fe700 1 -- 172.21.15.22:0/1990880694 learned_addr learned my addr 172.21.15.22:0/1990880694 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:46:33.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.469+0000 7f0ebbfff700 1 --2- 172.21.15.22:0/1990880694 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0ebc0ff9d0 0x7f0ebc1966a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:33.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.469+0000 7f0ebb7fe700 1 -- 172.21.15.22:0/1990880694 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0ebc0ff9d0 msgr2=0x7f0ebc1966a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:33.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.469+0000 7f0ebb7fe700 1 --2- 172.21.15.22:0/1990880694 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0ebc0ff9d0 0x7f0ebc1966a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:33.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.469+0000 7f0ebb7fe700 1 -- 172.21.15.22:0/1990880694 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb00095f0 con 0x7f0ebc100bd0 2024-03-24T09:46:33.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.469+0000 7f0ebbfff700 1 --2- 172.21.15.22:0/1990880694 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0ebc0ff9d0 0x7f0ebc1966a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:46:33.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.470+0000 7f0ebb7fe700 1 --2- 172.21.15.22:0/1990880694 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0ebc100bd0 0x7f0ebc196be0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0eb00059d0 tx=0x7f0eb000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:33.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.470+0000 7f0eb97fa700 1 -- 172.21.15.22:0/1990880694 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0eb001d070 con 0x7f0ebc100bd0 2024-03-24T09:46:33.473 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.470+0000 7f0ec23e6700 1 -- 172.21.15.22:0/1990880694 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ebc19bd40 con 0x7f0ebc100bd0 2024-03-24T09:46:33.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.470+0000 7f0eb97fa700 1 -- 172.21.15.22:0/1990880694 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0eb000f460 con 0x7f0ebc100bd0 2024-03-24T09:46:33.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.470+0000 7f0eb97fa700 1 -- 172.21.15.22:0/1990880694 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0eb0021640 con 0x7f0ebc100bd0 2024-03-24T09:46:33.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.470+0000 7f0ec23e6700 1 -- 172.21.15.22:0/1990880694 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ebc19c2b0 con 0x7f0ebc100bd0 2024-03-24T09:46:33.474 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.471+0000 7f0ec23e6700 1 -- 172.21.15.22:0/1990880694 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ebc04ea80 con 0x7f0ebc100bd0 2024-03-24T09:46:33.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.471+0000 7f0eb97fa700 1 -- 172.21.15.22:0/1990880694 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f0eb000f5d0 con 0x7f0ebc100bd0 2024-03-24T09:46:33.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.472+0000 7f0eb97fa700 1 --2- 172.21.15.22:0/1990880694 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f0ea406cce0 0x7f0ea406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:33.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.472+0000 7f0eb97fa700 1 -- 172.21.15.22:0/1990880694 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f0eb008d590 con 0x7f0ebc100bd0 2024-03-24T09:46:33.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.472+0000 7f0ebbfff700 1 --2- 172.21.15.22:0/1990880694 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f0ea406cce0 0x7f0ea406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:33.475 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.472+0000 7f0ebbfff700 1 --2- 172.21.15.22:0/1990880694 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f0ea406cce0 0x7f0ea406f1a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc100a30 tx=0x7f0eac008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:33.477 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.476+0000 7f0eb97fa700 1 -- 172.21.15.22:0/1990880694 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0eb005b3f0 con 0x7f0ebc100bd0 2024-03-24T09:46:33.731 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.731+0000 7f0ec23e6700 1 -- 172.21.15.22:0/1990880694 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0ebc19c560 con 0x7f0ebc100bd0 2024-03-24T09:46:33.732 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.731+0000 7f0eb97fa700 1 -- 172.21.15.22:0/1990880694 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1114 (secure 0 0 0) 0x7f0eb005af80 con 0x7f0ebc100bd0 2024-03-24T09:46:33.732 INFO:teuthology.orchestra.run.smithi022.stdout:e2 2024-03-24T09:46:33.732 INFO:teuthology.orchestra.run.smithi022.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T09:46:33.732 INFO:teuthology.orchestra.run.smithi022.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-24T09:46:33.732 INFO:teuthology.orchestra.run.smithi022.stdout:legacy client fscid: 1 2024-03-24T09:46:33.732 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:46:33.732 INFO:teuthology.orchestra.run.smithi022.stdout:Filesystem 'cephfs' (1) 2024-03-24T09:46:33.732 INFO:teuthology.orchestra.run.smithi022.stdout:fs_name cephfs 2024-03-24T09:46:33.733 INFO:teuthology.orchestra.run.smithi022.stdout:epoch 2 2024-03-24T09:46:33.733 INFO:teuthology.orchestra.run.smithi022.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-24T09:46:33.733 INFO:teuthology.orchestra.run.smithi022.stdout:created 2024-03-24T09:46:29.665102+0000 2024-03-24T09:46:33.733 INFO:teuthology.orchestra.run.smithi022.stdout:modified 2024-03-24T09:46:29.665196+0000 2024-03-24T09:46:33.733 INFO:teuthology.orchestra.run.smithi022.stdout:tableserver 0 2024-03-24T09:46:33.733 INFO:teuthology.orchestra.run.smithi022.stdout:root 0 2024-03-24T09:46:33.733 INFO:teuthology.orchestra.run.smithi022.stdout:session_timeout 60 2024-03-24T09:46:33.733 INFO:teuthology.orchestra.run.smithi022.stdout:session_autoclose 300 2024-03-24T09:46:33.733 INFO:teuthology.orchestra.run.smithi022.stdout:max_file_size 1099511627776 2024-03-24T09:46:33.733 INFO:teuthology.orchestra.run.smithi022.stdout:max_xattr_size 65536 2024-03-24T09:46:33.734 INFO:teuthology.orchestra.run.smithi022.stdout:required_client_features {} 2024-03-24T09:46:33.734 INFO:teuthology.orchestra.run.smithi022.stdout:last_failure 0 2024-03-24T09:46:33.734 INFO:teuthology.orchestra.run.smithi022.stdout:last_failure_osd_epoch 0 2024-03-24T09:46:33.734 INFO:teuthology.orchestra.run.smithi022.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-24T09:46:33.734 INFO:teuthology.orchestra.run.smithi022.stdout:max_mds 1 2024-03-24T09:46:33.734 INFO:teuthology.orchestra.run.smithi022.stdout:in 2024-03-24T09:46:33.734 INFO:teuthology.orchestra.run.smithi022.stdout:up {} 2024-03-24T09:46:33.734 INFO:teuthology.orchestra.run.smithi022.stdout:failed 2024-03-24T09:46:33.734 INFO:teuthology.orchestra.run.smithi022.stdout:damaged 2024-03-24T09:46:33.735 INFO:teuthology.orchestra.run.smithi022.stdout:stopped 2024-03-24T09:46:33.735 INFO:teuthology.orchestra.run.smithi022.stdout:data_pools [3] 2024-03-24T09:46:33.735 INFO:teuthology.orchestra.run.smithi022.stdout:metadata_pool 2 2024-03-24T09:46:33.735 INFO:teuthology.orchestra.run.smithi022.stdout:inline_data disabled 2024-03-24T09:46:33.735 INFO:teuthology.orchestra.run.smithi022.stdout:balancer 2024-03-24T09:46:33.735 INFO:teuthology.orchestra.run.smithi022.stdout:bal_rank_mask -1 2024-03-24T09:46:33.735 INFO:teuthology.orchestra.run.smithi022.stdout:standby_count_wanted 0 2024-03-24T09:46:33.735 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:46:33.735 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:46:33.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.733+0000 7f0ec23e6700 1 -- 172.21.15.22:0/1990880694 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f0ea406cce0 msgr2=0x7f0ea406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:33.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.733+0000 7f0ec23e6700 1 --2- 172.21.15.22:0/1990880694 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f0ea406cce0 0x7f0ea406f1a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc100a30 tx=0x7f0eac008040 comp rx=0 tx=0).stop 2024-03-24T09:46:33.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.733+0000 7f0ec23e6700 1 -- 172.21.15.22:0/1990880694 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0ebc100bd0 msgr2=0x7f0ebc196be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:33.737 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.733+0000 7f0ec23e6700 1 --2- 172.21.15.22:0/1990880694 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0ebc100bd0 0x7f0ebc196be0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0eb00059d0 tx=0x7f0eb000be20 comp rx=0 tx=0).stop 2024-03-24T09:46:33.737 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.734+0000 7f0ec23e6700 1 -- 172.21.15.22:0/1990880694 shutdown_connections 2024-03-24T09:46:33.737 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.734+0000 7f0ec23e6700 1 --2- 172.21.15.22:0/1990880694 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f0ea406cce0 0x7f0ea406f1a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:33.737 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.734+0000 7f0ec23e6700 1 --2- 172.21.15.22:0/1990880694 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0ebc0ff9d0 0x7f0ebc1966a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:33.737 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.734+0000 7f0ec23e6700 1 --2- 172.21.15.22:0/1990880694 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0ebc100bd0 0x7f0ebc196be0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:33.737 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.734+0000 7f0ec23e6700 1 -- 172.21.15.22:0/1990880694 >> 172.21.15.22:0/1990880694 conn(0x7f0ebc0faf70 msgr2=0x7f0ebc068510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:33.737 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.734+0000 7f0ec23e6700 1 -- 172.21.15.22:0/1990880694 shutdown_connections 2024-03-24T09:46:33.737 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:33.734+0000 7f0ec23e6700 1 -- 172.21.15.22:0/1990880694 wait complete. 2024-03-24T09:46:33.737 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 2 2024-03-24T09:46:33.852 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:33 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/1990880694' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-24T09:46:34.211 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:33 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/1990880694' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-24T09:46:34.628 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T09:46:34.640 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi022.front.sepia.ceph.com 2024-03-24T09:46:34.640 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- bash -c 'ceph fs set cephfs max_mds 1' 2024-03-24T09:46:34.860 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:46:35.211 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:34 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:35.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:34 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:35.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:34 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:35.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:34 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi155.pwhqcw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:46:35.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:34 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi155.pwhqcw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:46:35.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:34 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:46:35.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:34 smithi155 ceph-mon[37016]: Deploying daemon mds.cephfs.smithi155.pwhqcw on smithi155 2024-03-24T09:46:35.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:34 smithi155 ceph-mon[37016]: pgmap v181: 65 pgs: 50 active+clean, 9 creating+peering, 6 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:46:35.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:34 smithi155 ceph-mon[37016]: mds.? [v2:172.21.15.22:6826/3295717728,v1:172.21.15.22:6827/3295717728] up:boot 2024-03-24T09:46:35.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:34 smithi155 ceph-mon[37016]: daemon mds.cephfs.smithi022.mhcbrm assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-03-24T09:46:35.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:34 smithi155 ceph-mon[37016]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-24T09:46:35.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:34 smithi155 ceph-mon[37016]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-24T09:46:35.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:34 smithi155 ceph-mon[37016]: Cluster is now healthy 2024-03-24T09:46:35.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:34 smithi155 ceph-mon[37016]: fsmap cephfs:0 1 up:standby 2024-03-24T09:46:35.213 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:34 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi022.mhcbrm"}]: dispatch 2024-03-24T09:46:35.213 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:34 smithi155 ceph-mon[37016]: fsmap cephfs:1 {0=cephfs.smithi022.mhcbrm=up:creating} 2024-03-24T09:46:35.213 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:34 smithi155 ceph-mon[37016]: daemon mds.cephfs.smithi022.mhcbrm is now active in filesystem cephfs as rank 0 2024-03-24T09:46:35.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:34 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:35.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:34 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:35.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:34 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:35.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:34 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi155.pwhqcw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:46:35.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:34 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi155.pwhqcw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:46:35.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:34 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:46:35.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:34 smithi022 ceph-mon[29512]: Deploying daemon mds.cephfs.smithi155.pwhqcw on smithi155 2024-03-24T09:46:35.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:34 smithi022 ceph-mon[29512]: pgmap v181: 65 pgs: 50 active+clean, 9 creating+peering, 6 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:46:35.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:34 smithi022 ceph-mon[29512]: mds.? [v2:172.21.15.22:6826/3295717728,v1:172.21.15.22:6827/3295717728] up:boot 2024-03-24T09:46:35.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:34 smithi022 ceph-mon[29512]: daemon mds.cephfs.smithi022.mhcbrm assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-03-24T09:46:35.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:34 smithi022 ceph-mon[29512]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-24T09:46:35.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:34 smithi022 ceph-mon[29512]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-24T09:46:35.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:34 smithi022 ceph-mon[29512]: Cluster is now healthy 2024-03-24T09:46:35.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:34 smithi022 ceph-mon[29512]: fsmap cephfs:0 1 up:standby 2024-03-24T09:46:35.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:34 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi022.mhcbrm"}]: dispatch 2024-03-24T09:46:35.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:34 smithi022 ceph-mon[29512]: fsmap cephfs:1 {0=cephfs.smithi022.mhcbrm=up:creating} 2024-03-24T09:46:35.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:34 smithi022 ceph-mon[29512]: daemon mds.cephfs.smithi022.mhcbrm is now active in filesystem cephfs as rank 0 2024-03-24T09:46:36.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.475+0000 7f06bbf45700 1 -- 172.21.15.22:0/900796314 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f06b40ff360 msgr2=0x7f06b4101790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:36.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.475+0000 7f06bbf45700 1 --2- 172.21.15.22:0/900796314 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f06b40ff360 0x7f06b4101790 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f06ac00b0a0 tx=0x7f06ac00b3b0 comp rx=0 tx=0).stop 2024-03-24T09:46:36.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.475+0000 7f06bbf45700 1 -- 172.21.15.22:0/900796314 shutdown_connections 2024-03-24T09:46:36.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.475+0000 7f06bbf45700 1 --2- 172.21.15.22:0/900796314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06b4101cd0 0x7f06b4104160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:36.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.475+0000 7f06bbf45700 1 --2- 172.21.15.22:0/900796314 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f06b40ff360 0x7f06b4101790 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:36.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.475+0000 7f06bbf45700 1 -- 172.21.15.22:0/900796314 >> 172.21.15.22:0/900796314 conn(0x7f06b40faf80 msgr2=0x7f06b40fd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:36.478 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.475+0000 7f06bbf45700 1 -- 172.21.15.22:0/900796314 shutdown_connections 2024-03-24T09:46:36.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.475+0000 7f06bbf45700 1 -- 172.21.15.22:0/900796314 wait complete. 2024-03-24T09:46:36.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.476+0000 7f06bbf45700 1 Processor -- start 2024-03-24T09:46:36.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.476+0000 7f06bbf45700 1 -- start start 2024-03-24T09:46:36.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.476+0000 7f06bbf45700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f06b40ff360 0x7f06b419cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:36.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.476+0000 7f06bbf45700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06b4101cd0 0x7f06b419d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:36.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.476+0000 7f06bbf45700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b419d950 con 0x7f06b40ff360 2024-03-24T09:46:36.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.476+0000 7f06bbf45700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b41a2310 con 0x7f06b4101cd0 2024-03-24T09:46:36.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.476+0000 7f06b9ce1700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f06b40ff360 0x7f06b419cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:36.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.476+0000 7f06b9ce1700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f06b40ff360 0x7f06b419cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:51530/0 (socket says 172.21.15.22:51530) 2024-03-24T09:46:36.479 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.476+0000 7f06b9ce1700 1 -- 172.21.15.22:0/3207468285 learned_addr learned my addr 172.21.15.22:0/3207468285 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:46:36.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.476+0000 7f06b94e0700 1 --2- 172.21.15.22:0/3207468285 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06b4101cd0 0x7f06b419d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:36.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.476+0000 7f06b9ce1700 1 -- 172.21.15.22:0/3207468285 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06b4101cd0 msgr2=0x7f06b419d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:36.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.476+0000 7f06b9ce1700 1 --2- 172.21.15.22:0/3207468285 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06b4101cd0 0x7f06b419d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:36.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.476+0000 7f06b9ce1700 1 -- 172.21.15.22:0/3207468285 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06ac009d00 con 0x7f06b40ff360 2024-03-24T09:46:36.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.476+0000 7f06b9ce1700 1 --2- 172.21.15.22:0/3207468285 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f06b40ff360 0x7f06b419cdf0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f06ac006130 tx=0x7f06ac003c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:36.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.477+0000 7f06b2ffd700 1 -- 172.21.15.22:0/3207468285 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06ac021070 con 0x7f06b40ff360 2024-03-24T09:46:36.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.477+0000 7f06bbf45700 1 -- 172.21.15.22:0/3207468285 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06b41a24b0 con 0x7f06b40ff360 2024-03-24T09:46:36.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.477+0000 7f06b2ffd700 1 -- 172.21.15.22:0/3207468285 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f06ac00ce60 con 0x7f06b40ff360 2024-03-24T09:46:36.480 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.477+0000 7f06b2ffd700 1 -- 172.21.15.22:0/3207468285 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06ac00e040 con 0x7f06b40ff360 2024-03-24T09:46:36.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.477+0000 7f06bbf45700 1 -- 172.21.15.22:0/3207468285 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06b41a2970 con 0x7f06b40ff360 2024-03-24T09:46:36.481 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.477+0000 7f06bbf45700 1 -- 172.21.15.22:0/3207468285 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06b4196c50 con 0x7f06b40ff360 2024-03-24T09:46:36.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.478+0000 7f06b2ffd700 1 -- 172.21.15.22:0/3207468285 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f06ac004250 con 0x7f06b40ff360 2024-03-24T09:46:36.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.479+0000 7f06b2ffd700 1 --2- 172.21.15.22:0/3207468285 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f069c06cce0 0x7f069c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:36.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.479+0000 7f06b2ffd700 1 -- 172.21.15.22:0/3207468285 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f06ac08d600 con 0x7f06b40ff360 2024-03-24T09:46:36.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.479+0000 7f06b94e0700 1 --2- 172.21.15.22:0/3207468285 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f069c06cce0 0x7f069c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:36.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.479+0000 7f06b94e0700 1 --2- 172.21.15.22:0/3207468285 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f069c06cce0 0x7f069c06f1a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f06a8007c60 tx=0x7f06a80073d0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:36.484 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.483+0000 7f06b2ffd700 1 -- 172.21.15.22:0/3207468285 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f06ac0579d0 con 0x7f06b40ff360 2024-03-24T09:46:36.736 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.736+0000 7f06bbf45700 1 -- 172.21.15.22:0/3207468285 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7f06b402ce30 con 0x7f06b40ff360 2024-03-24T09:46:36.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.796+0000 7f06b2ffd700 1 -- 172.21.15.22:0/3207468285 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v6) v1 ==== 105+0+0 (secure 0 0 0) 0x7f06ac05aff0 con 0x7f06b40ff360 2024-03-24T09:46:36.799 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.798+0000 7f06bbf45700 1 -- 172.21.15.22:0/3207468285 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f069c06cce0 msgr2=0x7f069c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:36.799 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.798+0000 7f06bbf45700 1 --2- 172.21.15.22:0/3207468285 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f069c06cce0 0x7f069c06f1a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f06a8007c60 tx=0x7f06a80073d0 comp rx=0 tx=0).stop 2024-03-24T09:46:36.799 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.799+0000 7f06bbf45700 1 -- 172.21.15.22:0/3207468285 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f06b40ff360 msgr2=0x7f06b419cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:36.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.799+0000 7f06bbf45700 1 --2- 172.21.15.22:0/3207468285 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f06b40ff360 0x7f06b419cdf0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f06ac006130 tx=0x7f06ac003c30 comp rx=0 tx=0).stop 2024-03-24T09:46:36.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.799+0000 7f06bbf45700 1 -- 172.21.15.22:0/3207468285 shutdown_connections 2024-03-24T09:46:36.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.799+0000 7f06bbf45700 1 --2- 172.21.15.22:0/3207468285 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f069c06cce0 0x7f069c06f1a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:36.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.799+0000 7f06bbf45700 1 --2- 172.21.15.22:0/3207468285 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f06b40ff360 0x7f06b419cdf0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:36.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.799+0000 7f06bbf45700 1 --2- 172.21.15.22:0/3207468285 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06b4101cd0 0x7f06b419d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:36.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.799+0000 7f06bbf45700 1 -- 172.21.15.22:0/3207468285 >> 172.21.15.22:0/3207468285 conn(0x7f06b40faf80 msgr2=0x7f06b40fd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:36.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.799+0000 7f06bbf45700 1 -- 172.21.15.22:0/3207468285 shutdown_connections 2024-03-24T09:46:36.800 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:36.799+0000 7f06bbf45700 1 -- 172.21.15.22:0/3207468285 wait complete. 2024-03-24T09:46:37.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:36 smithi022 ceph-mon[29512]: mds.? [v2:172.21.15.22:6826/3295717728,v1:172.21.15.22:6827/3295717728] up:active 2024-03-24T09:46:37.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:36 smithi022 ceph-mon[29512]: fsmap cephfs:1 {0=cephfs.smithi022.mhcbrm=up:active} 2024-03-24T09:46:37.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:36 smithi022 ceph-mon[29512]: pgmap v182: 65 pgs: 65 active+clean; 578 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 449 B/s wr, 1 op/s 2024-03-24T09:46:37.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:36 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:37.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:36 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:37.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:36 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:37.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:36 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi022.qbmomh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:46:37.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:36 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi022.qbmomh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:46:37.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:36 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:46:37.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:36 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/3207468285' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-03-24T09:46:37.211 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:36 smithi155 ceph-mon[37016]: mds.? [v2:172.21.15.22:6826/3295717728,v1:172.21.15.22:6827/3295717728] up:active 2024-03-24T09:46:37.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:36 smithi155 ceph-mon[37016]: fsmap cephfs:1 {0=cephfs.smithi022.mhcbrm=up:active} 2024-03-24T09:46:37.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:36 smithi155 ceph-mon[37016]: pgmap v182: 65 pgs: 65 active+clean; 578 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 449 B/s wr, 1 op/s 2024-03-24T09:46:37.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:36 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:37.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:36 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:37.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:36 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:37.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:36 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi022.qbmomh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:46:37.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:36 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi022.qbmomh", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:46:37.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:36 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:46:37.213 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:36 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/3207468285' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-03-24T09:46:37.472 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T09:46:37.482 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi022.front.sepia.ceph.com 2024-03-24T09:46:37.482 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- bash -c 'ceph fs set cephfs allow_standby_replay true' 2024-03-24T09:46:37.897 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:37 smithi155 ceph-mon[37016]: Deploying daemon mds.cephfs.smithi022.qbmomh on smithi022 2024-03-24T09:46:37.897 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:37 smithi155 ceph-mon[37016]: mds.? [v2:172.21.15.155:6824/4090729184,v1:172.21.15.155:6825/4090729184] up:boot 2024-03-24T09:46:37.897 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:37 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/3207468285' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-03-24T09:46:37.898 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:37 smithi155 ceph-mon[37016]: fsmap cephfs:1 {0=cephfs.smithi022.mhcbrm=up:active} 1 up:standby 2024-03-24T09:46:37.898 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:37 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi155.pwhqcw"}]: dispatch 2024-03-24T09:46:37.898 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:37 smithi155 ceph-mon[37016]: fsmap cephfs:1 {0=cephfs.smithi022.mhcbrm=up:active} 1 up:standby 2024-03-24T09:46:38.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:37 smithi022 ceph-mon[29512]: Deploying daemon mds.cephfs.smithi022.qbmomh on smithi022 2024-03-24T09:46:38.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:37 smithi022 ceph-mon[29512]: mds.? [v2:172.21.15.155:6824/4090729184,v1:172.21.15.155:6825/4090729184] up:boot 2024-03-24T09:46:38.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:37 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/3207468285' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-03-24T09:46:38.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:37 smithi022 ceph-mon[29512]: fsmap cephfs:1 {0=cephfs.smithi022.mhcbrm=up:active} 1 up:standby 2024-03-24T09:46:38.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:37 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi155.pwhqcw"}]: dispatch 2024-03-24T09:46:38.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:37 smithi022 ceph-mon[29512]: fsmap cephfs:1 {0=cephfs.smithi022.mhcbrm=up:active} 1 up:standby 2024-03-24T09:46:38.629 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:46:39.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:38 smithi022 ceph-mon[29512]: pgmap v183: 65 pgs: 65 active+clean; 578 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 767 B/s wr, 4 op/s 2024-03-24T09:46:39.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:38 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:46:39.211 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:38 smithi155 ceph-mon[37016]: pgmap v183: 65 pgs: 65 active+clean; 578 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 767 B/s wr, 4 op/s 2024-03-24T09:46:39.211 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:38 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:46:40.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:40.413+0000 7f6d06006700 1 -- 172.21.15.22:0/905756424 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6d00100be0 msgr2=0x7f6d00101040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:40.416 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:40.413+0000 7f6d06006700 1 --2- 172.21.15.22:0/905756424 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6d00100be0 0x7f6d00101040 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4009910 tx=0x7f6cf4009c20 comp rx=0 tx=0).stop 2024-03-24T09:46:40.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:40.413+0000 7f6d06006700 1 -- 172.21.15.22:0/905756424 shutdown_connections 2024-03-24T09:46:40.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:40.413+0000 7f6d06006700 1 --2- 172.21.15.22:0/905756424 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6d00100be0 0x7f6d00101040 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:40.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:40.413+0000 7f6d06006700 1 --2- 172.21.15.22:0/905756424 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d000ff9e0 0x7f6d000ffe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:40.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:40.413+0000 7f6d06006700 1 -- 172.21.15.22:0/905756424 >> 172.21.15.22:0/905756424 conn(0x7f6d000faf80 msgr2=0x7f6d000fd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:40.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:40.413+0000 7f6d06006700 1 -- 172.21.15.22:0/905756424 shutdown_connections 2024-03-24T09:46:40.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:40.413+0000 7f6d06006700 1 -- 172.21.15.22:0/905756424 wait complete. 2024-03-24T09:46:40.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:40.414+0000 7f6d06006700 1 Processor -- start 2024-03-24T09:46:40.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:40.414+0000 7f6d06006700 1 -- start start 2024-03-24T09:46:40.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:40.414+0000 7f6d06006700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6d000ff9e0 0x7f6d00196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:40.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:40.414+0000 7f6d06006700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d00100be0 0x7f6d00196bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:40.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:40.414+0000 7f6d06006700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d001971f0 con 0x7f6d000ff9e0 2024-03-24T09:46:40.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:40.414+0000 7f6d06006700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d00197330 con 0x7f6d00100be0 2024-03-24T09:46:40.417 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:40.415+0000 7f6cff7fe700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6d000ff9e0 0x7f6d00196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:40.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:40.415+0000 7f6cff7fe700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6d000ff9e0 0x7f6d00196690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:48590/0 (socket says 172.21.15.22:48590) 2024-03-24T09:46:40.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:40.415+0000 7f6cff7fe700 1 -- 172.21.15.22:0/2834517699 learned_addr learned my addr 172.21.15.22:0/2834517699 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:46:40.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:40.415+0000 7f6cfeffd700 1 --2- 172.21.15.22:0/2834517699 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d00100be0 0x7f6d00196bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:40.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:40.415+0000 7f6cff7fe700 1 -- 172.21.15.22:0/2834517699 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d00100be0 msgr2=0x7f6d00196bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:40.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:40.415+0000 7f6cff7fe700 1 --2- 172.21.15.22:0/2834517699 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d00100be0 0x7f6d00196bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:40.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:40.415+0000 7f6cff7fe700 1 -- 172.21.15.22:0/2834517699 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf40095f0 con 0x7f6d000ff9e0 2024-03-24T09:46:40.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:40.415+0000 7f6cff7fe700 1 --2- 172.21.15.22:0/2834517699 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6d000ff9e0 0x7f6d00196690 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6cf000b480 tx=0x7f6cf000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:40.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:40.415+0000 7f6cfcff9700 1 -- 172.21.15.22:0/2834517699 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6cf00118c0 con 0x7f6d000ff9e0 2024-03-24T09:46:40.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:40.415+0000 7f6d06006700 1 -- 172.21.15.22:0/2834517699 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d0019bd40 con 0x7f6d000ff9e0 2024-03-24T09:46:40.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:40.415+0000 7f6cfcff9700 1 -- 172.21.15.22:0/2834517699 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6cf0010380 con 0x7f6d000ff9e0 2024-03-24T09:46:40.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:40.416+0000 7f6cfcff9700 1 -- 172.21.15.22:0/2834517699 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6cf000f4f0 con 0x7f6d000ff9e0 2024-03-24T09:46:40.418 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:40.416+0000 7f6d06006700 1 -- 172.21.15.22:0/2834517699 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d0019c2b0 con 0x7f6d000ff9e0 2024-03-24T09:46:40.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:40.417+0000 7f6d06006700 1 -- 172.21.15.22:0/2834517699 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d0004eaf0 con 0x7f6d000ff9e0 2024-03-24T09:46:40.419 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:40.417+0000 7f6cfcff9700 1 -- 172.21.15.22:0/2834517699 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f6cf0011a20 con 0x7f6d000ff9e0 2024-03-24T09:46:40.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:40.417+0000 7f6cfcff9700 1 --2- 172.21.15.22:0/2834517699 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f6ce806ccf0 0x7f6ce806f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:40.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:40.417+0000 7f6cfcff9700 1 -- 172.21.15.22:0/2834517699 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f6cf008be30 con 0x7f6d000ff9e0 2024-03-24T09:46:40.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:40.417+0000 7f6cfeffd700 1 --2- 172.21.15.22:0/2834517699 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f6ce806ccf0 0x7f6ce806f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:40.420 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:40.418+0000 7f6cfeffd700 1 --2- 172.21.15.22:0/2834517699 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f6ce806ccf0 0x7f6ce806f1b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6cf40051c0 tx=0x7f6cf400b540 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:40.422 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:40.422+0000 7f6cfcff9700 1 -- 172.21.15.22:0/2834517699 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6d0004eaf0 con 0x7f6d000ff9e0 2024-03-24T09:46:40.674 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:40.674+0000 7f6d06006700 1 -- 172.21.15.22:0/2834517699 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f6d0019c910 con 0x7f6d000ff9e0 2024-03-24T09:46:41.157 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:41.155+0000 7f6cfcff9700 1 -- 172.21.15.22:0/2834517699 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v8) v1 ==== 121+0+0 (secure 0 0 0) 0x7f6cf0059d10 con 0x7f6d000ff9e0 2024-03-24T09:46:41.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:41.158+0000 7f6d06006700 1 -- 172.21.15.22:0/2834517699 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f6ce806ccf0 msgr2=0x7f6ce806f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:41.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:41.158+0000 7f6d06006700 1 --2- 172.21.15.22:0/2834517699 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f6ce806ccf0 0x7f6ce806f1b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6cf40051c0 tx=0x7f6cf400b540 comp rx=0 tx=0).stop 2024-03-24T09:46:41.158 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:41.158+0000 7f6d06006700 1 -- 172.21.15.22:0/2834517699 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6d000ff9e0 msgr2=0x7f6d00196690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:41.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:41.158+0000 7f6d06006700 1 --2- 172.21.15.22:0/2834517699 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6d000ff9e0 0x7f6d00196690 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6cf000b480 tx=0x7f6cf000b840 comp rx=0 tx=0).stop 2024-03-24T09:46:41.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:41.158+0000 7f6d06006700 1 -- 172.21.15.22:0/2834517699 shutdown_connections 2024-03-24T09:46:41.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:41.158+0000 7f6d06006700 1 --2- 172.21.15.22:0/2834517699 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f6ce806ccf0 0x7f6ce806f1b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:41.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:41.158+0000 7f6d06006700 1 --2- 172.21.15.22:0/2834517699 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6d000ff9e0 0x7f6d00196690 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:41.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:41.158+0000 7f6d06006700 1 --2- 172.21.15.22:0/2834517699 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d00100be0 0x7f6d00196bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:41.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:41.158+0000 7f6d06006700 1 -- 172.21.15.22:0/2834517699 >> 172.21.15.22:0/2834517699 conn(0x7f6d000faf80 msgr2=0x7f6d00069080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:41.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:41.158+0000 7f6d06006700 1 -- 172.21.15.22:0/2834517699 shutdown_connections 2024-03-24T09:46:41.159 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:41.158+0000 7f6d06006700 1 -- 172.21.15.22:0/2834517699 wait complete. 2024-03-24T09:46:41.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:41 smithi022 ceph-mon[29512]: pgmap v184: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-03-24T09:46:41.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:41 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/2834517699' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-24T09:46:41.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:41 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:41.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:41 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:41.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:41 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:41.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:41 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi155.dakojr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:46:41.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:41 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi155.dakojr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:46:41.283 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:41 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:46:41.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:41 smithi155 ceph-mon[37016]: pgmap v184: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-03-24T09:46:41.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:41 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/2834517699' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-24T09:46:41.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:41 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:41.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:41 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:41.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:41 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:41.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:41 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi155.dakojr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:46:41.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:41 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi155.dakojr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:46:41.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:41 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:46:41.823 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T09:46:41.834 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi022.front.sepia.ceph.com 2024-03-24T09:46:41.834 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- bash -c 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it' 2024-03-24T09:46:42.053 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:46:42.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:42 smithi155 ceph-mon[37016]: Deploying daemon mds.cephfs.smithi155.dakojr on smithi155 2024-03-24T09:46:42.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:42 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/2834517699' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-24T09:46:42.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:42 smithi155 ceph-mon[37016]: mds.? [v2:172.21.15.22:6828/2909541441,v1:172.21.15.22:6829/2909541441] up:boot 2024-03-24T09:46:42.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:42 smithi155 ceph-mon[37016]: fsmap cephfs:1 {0=cephfs.smithi022.mhcbrm=up:active} 2 up:standby 2024-03-24T09:46:42.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:42 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi022.qbmomh"}]: dispatch 2024-03-24T09:46:42.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:42 smithi155 ceph-mon[37016]: fsmap cephfs:1 {0=cephfs.smithi022.mhcbrm=up:active} 1 up:standby-replay 1 up:standby 2024-03-24T09:46:42.493 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:42 smithi022 ceph-mon[29512]: Deploying daemon mds.cephfs.smithi155.dakojr on smithi155 2024-03-24T09:46:42.493 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:42 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/2834517699' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-24T09:46:42.493 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:42 smithi022 ceph-mon[29512]: mds.? [v2:172.21.15.22:6828/2909541441,v1:172.21.15.22:6829/2909541441] up:boot 2024-03-24T09:46:42.493 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:42 smithi022 ceph-mon[29512]: fsmap cephfs:1 {0=cephfs.smithi022.mhcbrm=up:active} 2 up:standby 2024-03-24T09:46:42.493 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:42 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi022.qbmomh"}]: dispatch 2024-03-24T09:46:42.494 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:42 smithi022 ceph-mon[29512]: fsmap cephfs:1 {0=cephfs.smithi022.mhcbrm=up:active} 1 up:standby-replay 1 up:standby 2024-03-24T09:46:43.412 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:43 smithi155 ceph-mon[37016]: pgmap v185: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-03-24T09:46:43.442 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:43 smithi022 ceph-mon[29512]: pgmap v185: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-03-24T09:46:43.789 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:43.789+0000 7f8c4c867700 1 -- 172.21.15.22:0/3415217157 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8c480ffa00 msgr2=0x7f8c480ffe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:43.789 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:43.789+0000 7f8c4c867700 1 --2- 172.21.15.22:0/3415217157 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8c480ffa00 0x7f8c480ffe20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8c38009910 tx=0x7f8c38009c20 comp rx=0 tx=0).stop 2024-03-24T09:46:43.789 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:43.789+0000 7f8c4c867700 1 -- 172.21.15.22:0/3415217157 shutdown_connections 2024-03-24T09:46:43.789 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:43.789+0000 7f8c4c867700 1 --2- 172.21.15.22:0/3415217157 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c48100c00 0x7f8c48101060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:43.789 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:43.789+0000 7f8c4c867700 1 --2- 172.21.15.22:0/3415217157 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8c480ffa00 0x7f8c480ffe20 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:43.790 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:43.789+0000 7f8c4c867700 1 -- 172.21.15.22:0/3415217157 >> 172.21.15.22:0/3415217157 conn(0x7f8c480fafc0 msgr2=0x7f8c480fd3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:43.790 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:43.789+0000 7f8c4c867700 1 -- 172.21.15.22:0/3415217157 shutdown_connections 2024-03-24T09:46:43.790 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:43.789+0000 7f8c4c867700 1 -- 172.21.15.22:0/3415217157 wait complete. 2024-03-24T09:46:43.791 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:43.790+0000 7f8c4c867700 1 Processor -- start 2024-03-24T09:46:43.791 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:43.790+0000 7f8c4c867700 1 -- start start 2024-03-24T09:46:43.791 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:43.790+0000 7f8c4c867700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c480ffa00 0x7f8c48198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:43.791 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:43.790+0000 7f8c4c867700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8c48100c00 0x7f8c48198cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:43.791 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:43.790+0000 7f8c4c867700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c481992d0 con 0x7f8c48100c00 2024-03-24T09:46:43.791 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:43.790+0000 7f8c4c867700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c48199410 con 0x7f8c480ffa00 2024-03-24T09:46:43.792 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:43.790+0000 7f8c4659c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c480ffa00 0x7f8c48198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:43.792 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:43.790+0000 7f8c4659c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c480ffa00 0x7f8c48198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:40314/0 (socket says 172.21.15.22:40314) 2024-03-24T09:46:43.792 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:43.790+0000 7f8c4659c700 1 -- 172.21.15.22:0/2129773615 learned_addr learned my addr 172.21.15.22:0/2129773615 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:46:43.792 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:43.790+0000 7f8c45d9b700 1 --2- 172.21.15.22:0/2129773615 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8c48100c00 0x7f8c48198cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:43.792 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:43.790+0000 7f8c4659c700 1 -- 172.21.15.22:0/2129773615 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8c48100c00 msgr2=0x7f8c48198cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:43.792 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:43.790+0000 7f8c4659c700 1 --2- 172.21.15.22:0/2129773615 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8c48100c00 0x7f8c48198cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:43.792 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:43.790+0000 7f8c4659c700 1 -- 172.21.15.22:0/2129773615 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c380095f0 con 0x7f8c480ffa00 2024-03-24T09:46:43.792 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:43.791+0000 7f8c4659c700 1 --2- 172.21.15.22:0/2129773615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c480ffa00 0x7f8c48198770 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8c380059d0 tx=0x7f8c3800bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:43.792 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:43.791+0000 7f8c377fe700 1 -- 172.21.15.22:0/2129773615 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c3801d070 con 0x7f8c480ffa00 2024-03-24T09:46:43.792 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:43.791+0000 7f8c4c867700 1 -- 172.21.15.22:0/2129773615 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c4819dd70 con 0x7f8c480ffa00 2024-03-24T09:46:43.792 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:43.791+0000 7f8c377fe700 1 -- 172.21.15.22:0/2129773615 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8c38005dc0 con 0x7f8c480ffa00 2024-03-24T09:46:43.792 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:43.791+0000 7f8c377fe700 1 -- 172.21.15.22:0/2129773615 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c38021410 con 0x7f8c480ffa00 2024-03-24T09:46:43.792 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:43.791+0000 7f8c4c867700 1 -- 172.21.15.22:0/2129773615 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c4819e280 con 0x7f8c480ffa00 2024-03-24T09:46:43.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:43.792+0000 7f8c4c867700 1 -- 172.21.15.22:0/2129773615 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c4804eaf0 con 0x7f8c480ffa00 2024-03-24T09:46:43.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:43.792+0000 7f8c377fe700 1 -- 172.21.15.22:0/2129773615 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f8c3800f460 con 0x7f8c480ffa00 2024-03-24T09:46:43.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:43.793+0000 7f8c377fe700 1 --2- 172.21.15.22:0/2129773615 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f8c3006cce0 0x7f8c3006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:43.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:43.793+0000 7f8c377fe700 1 -- 172.21.15.22:0/2129773615 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f8c3808d450 con 0x7f8c480ffa00 2024-03-24T09:46:43.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:43.793+0000 7f8c45d9b700 1 --2- 172.21.15.22:0/2129773615 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f8c3006cce0 0x7f8c3006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:43.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:43.793+0000 7f8c45d9b700 1 --2- 172.21.15.22:0/2129773615 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f8c3006cce0 0x7f8c3006f1a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c005950 tx=0x7f8c3c00b410 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:43.802 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:43.802+0000 7f8c377fe700 1 -- 172.21.15.22:0/2129773615 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8c4804eaf0 con 0x7f8c480ffa00 2024-03-24T09:46:44.054 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:44.054+0000 7f8c4c867700 1 -- 172.21.15.22:0/2129773615 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8c4819e920 con 0x7f8c480ffa00 2024-03-24T09:46:44.163 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:44.163+0000 7f8c377fe700 1 -- 172.21.15.22:0/2129773615 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]=0 inline data enabled v10) v1 ==== 168+0+0 (secure 0 0 0) 0x7f8c3805b2b0 con 0x7f8c480ffa00 2024-03-24T09:46:44.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:44.166+0000 7f8c4c867700 1 -- 172.21.15.22:0/2129773615 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f8c3006cce0 msgr2=0x7f8c3006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:44.166 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:44.166+0000 7f8c4c867700 1 --2- 172.21.15.22:0/2129773615 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f8c3006cce0 0x7f8c3006f1a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c005950 tx=0x7f8c3c00b410 comp rx=0 tx=0).stop 2024-03-24T09:46:44.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:44.166+0000 7f8c4c867700 1 -- 172.21.15.22:0/2129773615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c480ffa00 msgr2=0x7f8c48198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:44.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:44.166+0000 7f8c4c867700 1 --2- 172.21.15.22:0/2129773615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c480ffa00 0x7f8c48198770 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8c380059d0 tx=0x7f8c3800bd70 comp rx=0 tx=0).stop 2024-03-24T09:46:44.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:44.166+0000 7f8c4c867700 1 -- 172.21.15.22:0/2129773615 shutdown_connections 2024-03-24T09:46:44.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:44.166+0000 7f8c4c867700 1 --2- 172.21.15.22:0/2129773615 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f8c3006cce0 0x7f8c3006f1a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:44.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:44.166+0000 7f8c4c867700 1 --2- 172.21.15.22:0/2129773615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c480ffa00 0x7f8c48198770 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:44.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:44.166+0000 7f8c4c867700 1 --2- 172.21.15.22:0/2129773615 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8c48100c00 0x7f8c48198cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:44.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:44.166+0000 7f8c4c867700 1 -- 172.21.15.22:0/2129773615 >> 172.21.15.22:0/2129773615 conn(0x7f8c480fafc0 msgr2=0x7f8c48069020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:44.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:44.166+0000 7f8c4c867700 1 -- 172.21.15.22:0/2129773615 shutdown_connections 2024-03-24T09:46:44.167 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:44.166+0000 7f8c4c867700 1 -- 172.21.15.22:0/2129773615 wait complete. 2024-03-24T09:46:44.167 INFO:teuthology.orchestra.run.smithi022.stderr:inline data enabled 2024-03-24T09:46:44.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:44 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:44.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:44 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:44.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:44 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:44.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:44 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:44.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:44 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:44.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:44 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:44.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:44 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:46:44.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:44 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/2129773615' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T09:46:44.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:44 smithi155 ceph-mon[37016]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T09:46:44.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:44 smithi155 ceph-mon[37016]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-03-24T09:46:44.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:44 smithi155 ceph-mon[37016]: mds.? [v2:172.21.15.155:6826/3192784756,v1:172.21.15.155:6827/3192784756] up:boot 2024-03-24T09:46:44.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:44 smithi155 ceph-mon[37016]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T09:46:44.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:44 smithi155 ceph-mon[37016]: fsmap cephfs:1 {0=cephfs.smithi022.mhcbrm=up:active} 1 up:standby-replay 2 up:standby 2024-03-24T09:46:44.712 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:44 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi155.dakojr"}]: dispatch 2024-03-24T09:46:44.753 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:44 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:44.753 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:44 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:44.753 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:44 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:44.753 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:44 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:44.753 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:44 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:44.753 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:44 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:44.754 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:44 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:46:44.754 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:44 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/2129773615' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T09:46:44.754 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:44 smithi022 ceph-mon[29512]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T09:46:44.754 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:44 smithi022 ceph-mon[29512]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-03-24T09:46:44.754 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:44 smithi022 ceph-mon[29512]: mds.? [v2:172.21.15.155:6826/3192784756,v1:172.21.15.155:6827/3192784756] up:boot 2024-03-24T09:46:44.754 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:44 smithi022 ceph-mon[29512]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T09:46:44.754 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:44 smithi022 ceph-mon[29512]: fsmap cephfs:1 {0=cephfs.smithi022.mhcbrm=up:active} 1 up:standby-replay 2 up:standby 2024-03-24T09:46:44.754 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:44 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi155.dakojr"}]: dispatch 2024-03-24T09:46:44.775 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T09:46:44.784 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi022.front.sepia.ceph.com 2024-03-24T09:46:44.785 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-24T09:46:45.462 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:46:45.707 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:45 smithi022 ceph-mon[29512]: pgmap v186: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 170 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-24T09:46:45.707 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:45 smithi022 ceph-mon[29512]: Dropping low affinity active daemon mds.cephfs.smithi022.mhcbrm in favor of higher affinity standby. 2024-03-24T09:46:45.707 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:45 smithi022 ceph-mon[29512]: Replacing daemon mds.cephfs.smithi022.mhcbrm as rank 0 with standby daemon mds.cephfs.smithi155.pwhqcw 2024-03-24T09:46:45.707 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:45 smithi022 ceph-mon[29512]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T09:46:45.708 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:45 smithi022 ceph-mon[29512]: osdmap e46: 6 total, 6 up, 6 in 2024-03-24T09:46:45.708 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:45 smithi022 ceph-mon[29512]: mds.? [v2:172.21.15.155:6824/4090729184,v1:172.21.15.155:6825/4090729184] up:standby-replay 2024-03-24T09:46:45.708 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:45 smithi022 ceph-mon[29512]: mds.? [v2:172.21.15.22:6828/2909541441,v1:172.21.15.22:6829/2909541441] up:standby 2024-03-24T09:46:45.708 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:45 smithi022 ceph-mon[29512]: fsmap cephfs:1/1 {0=cephfs.smithi155.pwhqcw=up:replay} 2 up:standby 2024-03-24T09:46:45.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:45 smithi155 ceph-mon[37016]: pgmap v186: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 170 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-24T09:46:45.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:45 smithi155 ceph-mon[37016]: Dropping low affinity active daemon mds.cephfs.smithi022.mhcbrm in favor of higher affinity standby. 2024-03-24T09:46:45.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:45 smithi155 ceph-mon[37016]: Replacing daemon mds.cephfs.smithi022.mhcbrm as rank 0 with standby daemon mds.cephfs.smithi155.pwhqcw 2024-03-24T09:46:45.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:45 smithi155 ceph-mon[37016]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T09:46:45.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:45 smithi155 ceph-mon[37016]: osdmap e46: 6 total, 6 up, 6 in 2024-03-24T09:46:45.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:45 smithi155 ceph-mon[37016]: mds.? [v2:172.21.15.155:6824/4090729184,v1:172.21.15.155:6825/4090729184] up:standby-replay 2024-03-24T09:46:45.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:45 smithi155 ceph-mon[37016]: mds.? [v2:172.21.15.22:6828/2909541441,v1:172.21.15.22:6829/2909541441] up:standby 2024-03-24T09:46:45.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:45 smithi155 ceph-mon[37016]: fsmap cephfs:1/1 {0=cephfs.smithi155.pwhqcw=up:replay} 2 up:standby 2024-03-24T09:46:46.925 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:46.923+0000 7f093eb9b700 1 -- 172.21.15.22:0/1574420059 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0938100bd0 msgr2=0x7f0938101030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:46.925 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:46.923+0000 7f093eb9b700 1 --2- 172.21.15.22:0/1574420059 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0938100bd0 0x7f0938101030 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f0924009910 tx=0x7f0924009c20 comp rx=0 tx=0).stop 2024-03-24T09:46:46.925 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:46.923+0000 7f093eb9b700 1 -- 172.21.15.22:0/1574420059 shutdown_connections 2024-03-24T09:46:46.925 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:46.924+0000 7f093eb9b700 1 --2- 172.21.15.22:0/1574420059 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0938100bd0 0x7f0938101030 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:46.925 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:46.924+0000 7f093eb9b700 1 --2- 172.21.15.22:0/1574420059 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09380ff9d0 0x7f09380ffdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:46.925 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:46.924+0000 7f093eb9b700 1 -- 172.21.15.22:0/1574420059 >> 172.21.15.22:0/1574420059 conn(0x7f09380faf70 msgr2=0x7f09380fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:46.925 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:46.924+0000 7f093eb9b700 1 -- 172.21.15.22:0/1574420059 shutdown_connections 2024-03-24T09:46:46.925 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:46.924+0000 7f093eb9b700 1 -- 172.21.15.22:0/1574420059 wait complete. 2024-03-24T09:46:46.925 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:46.924+0000 7f093eb9b700 1 Processor -- start 2024-03-24T09:46:46.925 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:46.924+0000 7f093eb9b700 1 -- start start 2024-03-24T09:46:46.925 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:46.924+0000 7f093eb9b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09380ff9d0 0x7f0938103910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:46.926 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:46.924+0000 7f093eb9b700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0938100bd0 0x7f0938101fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:46.926 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:46.924+0000 7f093eb9b700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09381024f0 con 0x7f0938100bd0 2024-03-24T09:46:46.926 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:46.924+0000 7f093eb9b700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0938102630 con 0x7f09380ff9d0 2024-03-24T09:46:46.926 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:46.924+0000 7f0937fff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0938100bd0 0x7f0938101fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:46.926 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:46.924+0000 7f0937fff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0938100bd0 0x7f0938101fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:48652/0 (socket says 172.21.15.22:48652) 2024-03-24T09:46:46.926 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:46.924+0000 7f0937fff700 1 -- 172.21.15.22:0/2470830968 learned_addr learned my addr 172.21.15.22:0/2470830968 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:46:46.926 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:46.925+0000 7f093c937700 1 --2- 172.21.15.22:0/2470830968 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09380ff9d0 0x7f0938103910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:46.926 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:46.925+0000 7f0937fff700 1 -- 172.21.15.22:0/2470830968 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09380ff9d0 msgr2=0x7f0938103910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:46.926 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:46.925+0000 7f0937fff700 1 --2- 172.21.15.22:0/2470830968 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09380ff9d0 0x7f0938103910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:46.926 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:46.925+0000 7f0937fff700 1 -- 172.21.15.22:0/2470830968 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09240095f0 con 0x7f0938100bd0 2024-03-24T09:46:46.927 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:46.925+0000 7f0937fff700 1 --2- 172.21.15.22:0/2470830968 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0938100bd0 0x7f0938101fb0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f0924006010 tx=0x7f092400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:46.927 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:46.925+0000 7f0935ffb700 1 -- 172.21.15.22:0/2470830968 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f092401d070 con 0x7f0938100bd0 2024-03-24T09:46:46.928 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:46.925+0000 7f093eb9b700 1 -- 172.21.15.22:0/2470830968 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09381028b0 con 0x7f0938100bd0 2024-03-24T09:46:46.928 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:46.925+0000 7f0935ffb700 1 -- 172.21.15.22:0/2470830968 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f092400f460 con 0x7f0938100bd0 2024-03-24T09:46:46.928 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:46.925+0000 7f0935ffb700 1 -- 172.21.15.22:0/2470830968 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0924021620 con 0x7f0938100bd0 2024-03-24T09:46:46.928 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:46.925+0000 7f093eb9b700 1 -- 172.21.15.22:0/2470830968 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0938102e20 con 0x7f0938100bd0 2024-03-24T09:46:46.928 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:46.927+0000 7f093eb9b700 1 -- 172.21.15.22:0/2470830968 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f093804ea80 con 0x7f0938100bd0 2024-03-24T09:46:46.928 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:46.927+0000 7f0935ffb700 1 -- 172.21.15.22:0/2470830968 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f092400f5d0 con 0x7f0938100bd0 2024-03-24T09:46:46.928 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:46.927+0000 7f0935ffb700 1 --2- 172.21.15.22:0/2470830968 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f092806cce0 0x7f092806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:46.928 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:46.928+0000 7f0935ffb700 1 -- 172.21.15.22:0/2470830968 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f092408d310 con 0x7f0938100bd0 2024-03-24T09:46:46.928 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:46.928+0000 7f093c937700 1 --2- 172.21.15.22:0/2470830968 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f092806cce0 0x7f092806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:46.928 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:46.928+0000 7f093c937700 1 --2- 172.21.15.22:0/2470830968 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f092806cce0 0x7f092806f1a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0938100a30 tx=0x7f092c009190 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:46.932 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:46.932+0000 7f0935ffb700 1 -- 172.21.15.22:0/2470830968 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f092405b190 con 0x7f0938100bd0 2024-03-24T09:46:47.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:46 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:47.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:46 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:47.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:46 smithi022 ceph-mon[29512]: mds.? [v2:172.21.15.155:6824/4090729184,v1:172.21.15.155:6825/4090729184] up:reconnect 2024-03-24T09:46:47.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:46 smithi022 ceph-mon[29512]: mds.? [v2:172.21.15.22:6826/3956813571,v1:172.21.15.22:6827/3956813571] up:boot 2024-03-24T09:46:47.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:46 smithi022 ceph-mon[29512]: fsmap cephfs:1/1 {0=cephfs.smithi155.pwhqcw=up:reconnect} 3 up:standby 2024-03-24T09:46:47.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:46 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi022.mhcbrm"}]: dispatch 2024-03-24T09:46:47.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:46 smithi022 ceph-mon[29512]: pgmap v188: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-03-24T09:46:47.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:47.187+0000 7f093eb9b700 1 -- 172.21.15.22:0/2470830968 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f093819e690 con 0x7f0938100bd0 2024-03-24T09:46:47.187 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:47.187+0000 7f0935ffb700 1 -- 172.21.15.22:0/2470830968 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 13 v13) v1 ==== 76+0+1857 (secure 0 0 0) 0x7f092405ad20 con 0x7f0938100bd0 2024-03-24T09:46:47.187 INFO:teuthology.orchestra.run.smithi022.stdout:e13 2024-03-24T09:46:47.188 INFO:teuthology.orchestra.run.smithi022.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T09:46:47.188 INFO:teuthology.orchestra.run.smithi022.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-24T09:46:47.188 INFO:teuthology.orchestra.run.smithi022.stdout:legacy client fscid: 1 2024-03-24T09:46:47.188 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:46:47.188 INFO:teuthology.orchestra.run.smithi022.stdout:Filesystem 'cephfs' (1) 2024-03-24T09:46:47.188 INFO:teuthology.orchestra.run.smithi022.stdout:fs_name cephfs 2024-03-24T09:46:47.188 INFO:teuthology.orchestra.run.smithi022.stdout:epoch 13 2024-03-24T09:46:47.188 INFO:teuthology.orchestra.run.smithi022.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-24T09:46:47.188 INFO:teuthology.orchestra.run.smithi022.stdout:created 2024-03-24T09:46:29.665102+0000 2024-03-24T09:46:47.188 INFO:teuthology.orchestra.run.smithi022.stdout:modified 2024-03-24T09:46:47.163379+0000 2024-03-24T09:46:47.188 INFO:teuthology.orchestra.run.smithi022.stdout:tableserver 0 2024-03-24T09:46:47.188 INFO:teuthology.orchestra.run.smithi022.stdout:root 0 2024-03-24T09:46:47.188 INFO:teuthology.orchestra.run.smithi022.stdout:session_timeout 60 2024-03-24T09:46:47.188 INFO:teuthology.orchestra.run.smithi022.stdout:session_autoclose 300 2024-03-24T09:46:47.188 INFO:teuthology.orchestra.run.smithi022.stdout:max_file_size 1099511627776 2024-03-24T09:46:47.189 INFO:teuthology.orchestra.run.smithi022.stdout:max_xattr_size 65536 2024-03-24T09:46:47.189 INFO:teuthology.orchestra.run.smithi022.stdout:required_client_features {} 2024-03-24T09:46:47.189 INFO:teuthology.orchestra.run.smithi022.stdout:last_failure 0 2024-03-24T09:46:47.189 INFO:teuthology.orchestra.run.smithi022.stdout:last_failure_osd_epoch 46 2024-03-24T09:46:47.189 INFO:teuthology.orchestra.run.smithi022.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-24T09:46:47.189 INFO:teuthology.orchestra.run.smithi022.stdout:max_mds 1 2024-03-24T09:46:47.189 INFO:teuthology.orchestra.run.smithi022.stdout:in 0 2024-03-24T09:46:47.189 INFO:teuthology.orchestra.run.smithi022.stdout:up {0=24269} 2024-03-24T09:46:47.189 INFO:teuthology.orchestra.run.smithi022.stdout:failed 2024-03-24T09:46:47.189 INFO:teuthology.orchestra.run.smithi022.stdout:damaged 2024-03-24T09:46:47.189 INFO:teuthology.orchestra.run.smithi022.stdout:stopped 2024-03-24T09:46:47.189 INFO:teuthology.orchestra.run.smithi022.stdout:data_pools [3] 2024-03-24T09:46:47.189 INFO:teuthology.orchestra.run.smithi022.stdout:metadata_pool 2 2024-03-24T09:46:47.189 INFO:teuthology.orchestra.run.smithi022.stdout:inline_data enabled 2024-03-24T09:46:47.189 INFO:teuthology.orchestra.run.smithi022.stdout:balancer 2024-03-24T09:46:47.189 INFO:teuthology.orchestra.run.smithi022.stdout:bal_rank_mask -1 2024-03-24T09:46:47.190 INFO:teuthology.orchestra.run.smithi022.stdout:standby_count_wanted 1 2024-03-24T09:46:47.190 INFO:teuthology.orchestra.run.smithi022.stdout:[mds.cephfs.smithi155.pwhqcw{0:24269} state up:rejoin seq 5 join_fscid=1 addr [v2:172.21.15.155:6824/4090729184,v1:172.21.15.155:6825/4090729184] compat {c=[1],r=[1],i=[7ff]}] 2024-03-24T09:46:47.190 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:46:47.190 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:46:47.190 INFO:teuthology.orchestra.run.smithi022.stdout:Standby daemons: 2024-03-24T09:46:47.190 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:46:47.190 INFO:teuthology.orchestra.run.smithi022.stdout:[mds.cephfs.smithi022.qbmomh{-1:14496} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.22:6828/2909541441,v1:172.21.15.22:6829/2909541441] compat {c=[1],r=[1],i=[7ff]}] 2024-03-24T09:46:47.190 INFO:teuthology.orchestra.run.smithi022.stdout:[mds.cephfs.smithi022.mhcbrm{-1:14504} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.22:6826/3956813571,v1:172.21.15.22:6827/3956813571] compat {c=[1],r=[1],i=[7ff]}] 2024-03-24T09:46:47.190 INFO:teuthology.orchestra.run.smithi022.stdout:[mds.cephfs.smithi155.dakojr{-1:24275} state up:standby seq 1 addr [v2:172.21.15.155:6826/3192784756,v1:172.21.15.155:6827/3192784756] compat {c=[1],r=[1],i=[7ff]}] 2024-03-24T09:46:47.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:47.189+0000 7f093eb9b700 1 -- 172.21.15.22:0/2470830968 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f092806cce0 msgr2=0x7f092806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:47.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:47.189+0000 7f093eb9b700 1 --2- 172.21.15.22:0/2470830968 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f092806cce0 0x7f092806f1a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0938100a30 tx=0x7f092c009190 comp rx=0 tx=0).stop 2024-03-24T09:46:47.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:47.189+0000 7f093eb9b700 1 -- 172.21.15.22:0/2470830968 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0938100bd0 msgr2=0x7f0938101fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:47.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:47.189+0000 7f093eb9b700 1 --2- 172.21.15.22:0/2470830968 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0938100bd0 0x7f0938101fb0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f0924006010 tx=0x7f092400bd70 comp rx=0 tx=0).stop 2024-03-24T09:46:47.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:47.189+0000 7f093eb9b700 1 -- 172.21.15.22:0/2470830968 shutdown_connections 2024-03-24T09:46:47.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:47.189+0000 7f093eb9b700 1 --2- 172.21.15.22:0/2470830968 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f092806cce0 0x7f092806f1a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:47.191 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:47.189+0000 7f093eb9b700 1 --2- 172.21.15.22:0/2470830968 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09380ff9d0 0x7f0938103910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:47.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:47.189+0000 7f093eb9b700 1 --2- 172.21.15.22:0/2470830968 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0938100bd0 0x7f0938101fb0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:47.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:47.189+0000 7f093eb9b700 1 -- 172.21.15.22:0/2470830968 >> 172.21.15.22:0/2470830968 conn(0x7f09380faf70 msgr2=0x7f0938068d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:47.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:47.190+0000 7f093eb9b700 1 -- 172.21.15.22:0/2470830968 shutdown_connections 2024-03-24T09:46:47.192 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:47.190+0000 7f093eb9b700 1 -- 172.21.15.22:0/2470830968 wait complete. 2024-03-24T09:46:47.192 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 13 2024-03-24T09:46:47.211 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:46 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:47.211 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:46 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:47.211 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:46 smithi155 ceph-mon[37016]: mds.? [v2:172.21.15.155:6824/4090729184,v1:172.21.15.155:6825/4090729184] up:reconnect 2024-03-24T09:46:47.211 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:46 smithi155 ceph-mon[37016]: mds.? [v2:172.21.15.22:6826/3956813571,v1:172.21.15.22:6827/3956813571] up:boot 2024-03-24T09:46:47.211 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:46 smithi155 ceph-mon[37016]: fsmap cephfs:1/1 {0=cephfs.smithi155.pwhqcw=up:reconnect} 3 up:standby 2024-03-24T09:46:47.211 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:46 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi022.mhcbrm"}]: dispatch 2024-03-24T09:46:47.212 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:46 smithi155 ceph-mon[37016]: pgmap v188: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-03-24T09:46:48.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:48 smithi155 ceph-mon[37016]: mds.? [v2:172.21.15.155:6824/4090729184,v1:172.21.15.155:6825/4090729184] up:rejoin 2024-03-24T09:46:48.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:48 smithi155 ceph-mon[37016]: fsmap cephfs:1/1 {0=cephfs.smithi155.pwhqcw=up:rejoin} 3 up:standby 2024-03-24T09:46:48.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:48 smithi155 ceph-mon[37016]: daemon mds.cephfs.smithi155.pwhqcw is now active in filesystem cephfs as rank 0 2024-03-24T09:46:48.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:48 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/2470830968' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-24T09:46:48.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:48 smithi022 ceph-mon[29512]: mds.? [v2:172.21.15.155:6824/4090729184,v1:172.21.15.155:6825/4090729184] up:rejoin 2024-03-24T09:46:48.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:48 smithi022 ceph-mon[29512]: fsmap cephfs:1/1 {0=cephfs.smithi155.pwhqcw=up:rejoin} 3 up:standby 2024-03-24T09:46:48.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:48 smithi022 ceph-mon[29512]: daemon mds.cephfs.smithi155.pwhqcw is now active in filesystem cephfs as rank 0 2024-03-24T09:46:48.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:48 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/2470830968' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-24T09:46:48.889 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-03-24T09:46:49.248 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:46:49.455 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:49 smithi022 ceph-mon[29512]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T09:46:49.455 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:49 smithi022 ceph-mon[29512]: mds.? [v2:172.21.15.155:6824/4090729184,v1:172.21.15.155:6825/4090729184] up:active 2024-03-24T09:46:49.455 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:49 smithi022 ceph-mon[29512]: mds.? [v2:172.21.15.155:6826/3192784756,v1:172.21.15.155:6827/3192784756] up:standby 2024-03-24T09:46:49.456 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:49 smithi022 ceph-mon[29512]: fsmap cephfs:1 {0=cephfs.smithi155.pwhqcw=up:active} 3 up:standby 2024-03-24T09:46:49.456 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:49 smithi022 ceph-mon[29512]: fsmap cephfs:1 {0=cephfs.smithi155.pwhqcw=up:active} 1 up:standby-replay 2 up:standby 2024-03-24T09:46:49.456 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:49 smithi022 ceph-mon[29512]: pgmap v189: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 819 B/s wr, 3 op/s 2024-03-24T09:46:49.456 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:49 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:49.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:49 smithi155 ceph-mon[37016]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T09:46:49.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:49 smithi155 ceph-mon[37016]: mds.? [v2:172.21.15.155:6824/4090729184,v1:172.21.15.155:6825/4090729184] up:active 2024-03-24T09:46:49.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:49 smithi155 ceph-mon[37016]: mds.? [v2:172.21.15.155:6826/3192784756,v1:172.21.15.155:6827/3192784756] up:standby 2024-03-24T09:46:49.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:49 smithi155 ceph-mon[37016]: fsmap cephfs:1 {0=cephfs.smithi155.pwhqcw=up:active} 3 up:standby 2024-03-24T09:46:49.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:49 smithi155 ceph-mon[37016]: fsmap cephfs:1 {0=cephfs.smithi155.pwhqcw=up:active} 1 up:standby-replay 2 up:standby 2024-03-24T09:46:49.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:49 smithi155 ceph-mon[37016]: pgmap v189: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 819 B/s wr, 3 op/s 2024-03-24T09:46:49.462 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:49 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:51.360 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:51 smithi022 ceph-mon[29512]: pgmap v190: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-24T09:46:51.366 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.360+0000 7f0afbab0700 1 -- 172.21.15.22:0/3910962355 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0af4100be0 msgr2=0x7f0af4101040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:51.367 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.360+0000 7f0afbab0700 1 --2- 172.21.15.22:0/3910962355 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0af4100be0 0x7f0af4101040 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0ae4009910 tx=0x7f0ae4009c20 comp rx=0 tx=0).stop 2024-03-24T09:46:51.367 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.360+0000 7f0afbab0700 1 -- 172.21.15.22:0/3910962355 shutdown_connections 2024-03-24T09:46:51.367 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.360+0000 7f0afbab0700 1 --2- 172.21.15.22:0/3910962355 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0af4100be0 0x7f0af4101040 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:51.367 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.360+0000 7f0afbab0700 1 --2- 172.21.15.22:0/3910962355 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0af40ff9e0 0x7f0af40ffe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:51.367 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.360+0000 7f0afbab0700 1 -- 172.21.15.22:0/3910962355 >> 172.21.15.22:0/3910962355 conn(0x7f0af40faf80 msgr2=0x7f0af40fd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:51.367 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.360+0000 7f0afbab0700 1 -- 172.21.15.22:0/3910962355 shutdown_connections 2024-03-24T09:46:51.367 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.360+0000 7f0afbab0700 1 -- 172.21.15.22:0/3910962355 wait complete. 2024-03-24T09:46:51.367 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.361+0000 7f0afbab0700 1 Processor -- start 2024-03-24T09:46:51.367 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.361+0000 7f0afbab0700 1 -- start start 2024-03-24T09:46:51.367 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.361+0000 7f0afbab0700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0af40ff9e0 0x7f0af41953d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:51.367 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.361+0000 7f0afbab0700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0af4100be0 0x7f0af4195910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:51.367 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.361+0000 7f0afbab0700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0af4195ea0 con 0x7f0af4100be0 2024-03-24T09:46:51.367 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.361+0000 7f0afbab0700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0af4195fe0 con 0x7f0af40ff9e0 2024-03-24T09:46:51.367 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.361+0000 7f0af904b700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0af4100be0 0x7f0af4195910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:51.368 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.361+0000 7f0af904b700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0af4100be0 0x7f0af4195910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:45146/0 (socket says 172.21.15.22:45146) 2024-03-24T09:46:51.368 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.361+0000 7f0af904b700 1 -- 172.21.15.22:0/2165395997 learned_addr learned my addr 172.21.15.22:0/2165395997 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:46:51.368 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.361+0000 7f0af984c700 1 --2- 172.21.15.22:0/2165395997 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0af40ff9e0 0x7f0af41953d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:51.368 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.361+0000 7f0af904b700 1 -- 172.21.15.22:0/2165395997 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0af40ff9e0 msgr2=0x7f0af41953d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:51.368 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.361+0000 7f0af904b700 1 --2- 172.21.15.22:0/2165395997 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0af40ff9e0 0x7f0af41953d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:51.368 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.362+0000 7f0af904b700 1 -- 172.21.15.22:0/2165395997 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae40095f0 con 0x7f0af4100be0 2024-03-24T09:46:51.368 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.362+0000 7f0af984c700 1 --2- 172.21.15.22:0/2165395997 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0af40ff9e0 0x7f0af41953d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T09:46:51.368 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.362+0000 7f0af904b700 1 --2- 172.21.15.22:0/2165395997 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0af4100be0 0x7f0af4195910 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0ae40092e0 tx=0x7f0ae400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:51.368 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.362+0000 7f0aeaffd700 1 -- 172.21.15.22:0/2165395997 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ae401d070 con 0x7f0af4100be0 2024-03-24T09:46:51.368 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.362+0000 7f0aeaffd700 1 -- 172.21.15.22:0/2165395997 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0ae400f460 con 0x7f0af4100be0 2024-03-24T09:46:51.368 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.362+0000 7f0afbab0700 1 -- 172.21.15.22:0/2165395997 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0af419a9f0 con 0x7f0af4100be0 2024-03-24T09:46:51.368 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.362+0000 7f0aeaffd700 1 -- 172.21.15.22:0/2165395997 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ae4021640 con 0x7f0af4100be0 2024-03-24T09:46:51.368 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.362+0000 7f0afbab0700 1 -- 172.21.15.22:0/2165395997 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0af419af30 con 0x7f0af4100be0 2024-03-24T09:46:51.368 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.363+0000 7f0afbab0700 1 -- 172.21.15.22:0/2165395997 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0af418f560 con 0x7f0af4100be0 2024-03-24T09:46:51.369 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.363+0000 7f0aeaffd700 1 -- 172.21.15.22:0/2165395997 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f0ae400f5d0 con 0x7f0af4100be0 2024-03-24T09:46:51.369 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.364+0000 7f0aeaffd700 1 --2- 172.21.15.22:0/2165395997 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f0ae006cce0 0x7f0ae006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:51.369 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.364+0000 7f0aeaffd700 1 -- 172.21.15.22:0/2165395997 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f0ae408d680 con 0x7f0af4100be0 2024-03-24T09:46:51.369 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.364+0000 7f0af984c700 1 --2- 172.21.15.22:0/2165395997 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f0ae006cce0 0x7f0ae006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:51.369 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.364+0000 7f0af984c700 1 --2- 172.21.15.22:0/2165395997 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f0ae006cce0 0x7f0ae006f1a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0af4100a40 tx=0x7f0af0009040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:51.369 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.368+0000 7f0aeaffd700 1 -- 172.21.15.22:0/2165395997 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0ae405b480 con 0x7f0af4100be0 2024-03-24T09:46:51.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:51 smithi155 ceph-mon[37016]: pgmap v190: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 3.6 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-24T09:46:51.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.618+0000 7f0afbab0700 1 -- 172.21.15.22:0/2165395997 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0af402cf30 con 0x7f0af4100be0 2024-03-24T09:46:51.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.619+0000 7f0aeaffd700 1 -- 172.21.15.22:0/2165395997 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4839 (secure 0 0 0) 0x7f0ae405b010 con 0x7f0af4100be0 2024-03-24T09:46:51.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.621+0000 7f0afbab0700 1 -- 172.21.15.22:0/2165395997 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f0ae006cce0 msgr2=0x7f0ae006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:51.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.621+0000 7f0afbab0700 1 --2- 172.21.15.22:0/2165395997 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f0ae006cce0 0x7f0ae006f1a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0af4100a40 tx=0x7f0af0009040 comp rx=0 tx=0).stop 2024-03-24T09:46:51.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.621+0000 7f0afbab0700 1 -- 172.21.15.22:0/2165395997 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0af4100be0 msgr2=0x7f0af4195910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:51.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.621+0000 7f0afbab0700 1 --2- 172.21.15.22:0/2165395997 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0af4100be0 0x7f0af4195910 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0ae40092e0 tx=0x7f0ae400be20 comp rx=0 tx=0).stop 2024-03-24T09:46:51.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.621+0000 7f0afbab0700 1 -- 172.21.15.22:0/2165395997 shutdown_connections 2024-03-24T09:46:51.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.621+0000 7f0afbab0700 1 --2- 172.21.15.22:0/2165395997 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f0ae006cce0 0x7f0ae006f1a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:51.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.621+0000 7f0afbab0700 1 --2- 172.21.15.22:0/2165395997 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0af40ff9e0 0x7f0af41953d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:51.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.621+0000 7f0afbab0700 1 --2- 172.21.15.22:0/2165395997 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0af4100be0 0x7f0af4195910 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:51.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.621+0000 7f0afbab0700 1 -- 172.21.15.22:0/2165395997 >> 172.21.15.22:0/2165395997 conn(0x7f0af40faf80 msgr2=0x7f0af4104230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:51.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.622+0000 7f0afbab0700 1 -- 172.21.15.22:0/2165395997 shutdown_connections 2024-03-24T09:46:51.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:51.622+0000 7f0afbab0700 1 -- 172.21.15.22:0/2165395997 wait complete. 2024-03-24T09:46:51.623 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 15 2024-03-24T09:46:51.632 INFO:teuthology.orchestra.run.smithi022.stdout:true 2024-03-24T09:46:52.171 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-03-24T09:46:52.396 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:46:52.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:52 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/2165395997' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:46:52.522 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:52 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/2165395997' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:46:53.733 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:53 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:53.733 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:53 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:53.733 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:53 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:46:53.733 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:53 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:46:53.733 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:53 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:53.734 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:53 smithi022 ceph-mon[29512]: pgmap v191: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 6.1 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T09:46:53.734 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:53 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:46:53.931 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:53 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:53.931 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:53 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:53.932 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:53 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:46:53.932 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:53 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:46:53.932 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:53 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:53.932 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:53 smithi155 ceph-mon[37016]: pgmap v191: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 6.1 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T09:46:53.932 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:53 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:46:54.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.094+0000 7f8970205700 1 -- 172.21.15.22:0/3587648273 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f89680ff9d0 msgr2=0x7f89680ffdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:54.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.094+0000 7f8970205700 1 --2- 172.21.15.22:0/3587648273 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f89680ff9d0 0x7f89680ffdf0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f8964009910 tx=0x7f8964009c20 comp rx=0 tx=0).stop 2024-03-24T09:46:54.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.094+0000 7f8970205700 1 -- 172.21.15.22:0/3587648273 shutdown_connections 2024-03-24T09:46:54.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.094+0000 7f8970205700 1 --2- 172.21.15.22:0/3587648273 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8968100bd0 0x7f8968101030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:54.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.094+0000 7f8970205700 1 --2- 172.21.15.22:0/3587648273 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f89680ff9d0 0x7f89680ffdf0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:54.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.094+0000 7f8970205700 1 -- 172.21.15.22:0/3587648273 >> 172.21.15.22:0/3587648273 conn(0x7f89680faf70 msgr2=0x7f89680fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:54.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.094+0000 7f8970205700 1 -- 172.21.15.22:0/3587648273 shutdown_connections 2024-03-24T09:46:54.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.094+0000 7f8970205700 1 -- 172.21.15.22:0/3587648273 wait complete. 2024-03-24T09:46:54.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.095+0000 7f8970205700 1 Processor -- start 2024-03-24T09:46:54.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.095+0000 7f8970205700 1 -- start start 2024-03-24T09:46:54.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.095+0000 7f8970205700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f89680ff9d0 0x7f8968196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:54.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.095+0000 7f8970205700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8968100bd0 0x7f8968196ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:54.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.095+0000 7f8970205700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89681970e0 con 0x7f8968100bd0 2024-03-24T09:46:54.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.095+0000 7f8970205700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8968197220 con 0x7f89680ff9d0 2024-03-24T09:46:54.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.095+0000 7f896dfa1700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f89680ff9d0 0x7f8968196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:54.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.095+0000 7f896dfa1700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f89680ff9d0 0x7f8968196580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:52366/0 (socket says 172.21.15.22:52366) 2024-03-24T09:46:54.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.095+0000 7f896dfa1700 1 -- 172.21.15.22:0/3907438941 learned_addr learned my addr 172.21.15.22:0/3907438941 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:46:54.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.095+0000 7f896d7a0700 1 --2- 172.21.15.22:0/3907438941 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8968100bd0 0x7f8968196ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:54.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.095+0000 7f896d7a0700 1 -- 172.21.15.22:0/3907438941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f89680ff9d0 msgr2=0x7f8968196580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:54.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.095+0000 7f896d7a0700 1 --2- 172.21.15.22:0/3907438941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f89680ff9d0 0x7f8968196580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:54.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.095+0000 7f896d7a0700 1 -- 172.21.15.22:0/3907438941 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89640095f0 con 0x7f8968100bd0 2024-03-24T09:46:54.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.095+0000 7f896dfa1700 1 --2- 172.21.15.22:0/3907438941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f89680ff9d0 0x7f8968196580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:46:54.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.096+0000 7f896d7a0700 1 --2- 172.21.15.22:0/3907438941 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8968100bd0 0x7f8968196ac0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f895800c8f0 tx=0x7f895800cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:54.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.096+0000 7f895effd700 1 -- 172.21.15.22:0/3907438941 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f895800e850 con 0x7f8968100bd0 2024-03-24T09:46:54.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.096+0000 7f895effd700 1 -- 172.21.15.22:0/3907438941 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f895800ee70 con 0x7f8968100bd0 2024-03-24T09:46:54.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.096+0000 7f8970205700 1 -- 172.21.15.22:0/3907438941 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f896819bc30 con 0x7f8968100bd0 2024-03-24T09:46:54.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.096+0000 7f895effd700 1 -- 172.21.15.22:0/3907438941 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8958018630 con 0x7f8968100bd0 2024-03-24T09:46:54.098 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.096+0000 7f8970205700 1 -- 172.21.15.22:0/3907438941 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f896819c200 con 0x7f8968100bd0 2024-03-24T09:46:54.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.097+0000 7f8970205700 1 -- 172.21.15.22:0/3907438941 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f896804ea80 con 0x7f8968100bd0 2024-03-24T09:46:54.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.097+0000 7f895effd700 1 -- 172.21.15.22:0/3907438941 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f895800f450 con 0x7f8968100bd0 2024-03-24T09:46:54.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.097+0000 7f895effd700 1 --2- 172.21.15.22:0/3907438941 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f895406cce0 0x7f895406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:54.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.098+0000 7f895effd700 1 -- 172.21.15.22:0/3907438941 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f895808c0e0 con 0x7f8968100bd0 2024-03-24T09:46:54.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.098+0000 7f896dfa1700 1 --2- 172.21.15.22:0/3907438941 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f895406cce0 0x7f895406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:54.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.098+0000 7f896dfa1700 1 --2- 172.21.15.22:0/3907438941 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f895406cce0 0x7f895406f1a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f89640051c0 tx=0x7f8964005960 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:54.102 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.102+0000 7f895effd700 1 -- 172.21.15.22:0/3907438941 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8958059ee0 con 0x7f8968100bd0 2024-03-24T09:46:54.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.391+0000 7f8970205700 1 -- 172.21.15.22:0/3907438941 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7f896819c450 con 0x7f8968100bd0 2024-03-24T09:46:54.391 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.391+0000 7f895effd700 1 -- 172.21.15.22:0/3907438941 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v15) v1 ==== 78+0+97 (secure 0 0 0) 0x7f895805ee60 con 0x7f8968100bd0 2024-03-24T09:46:54.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.393+0000 7f8970205700 1 -- 172.21.15.22:0/3907438941 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f895406cce0 msgr2=0x7f895406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:54.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.393+0000 7f8970205700 1 --2- 172.21.15.22:0/3907438941 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f895406cce0 0x7f895406f1a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f89640051c0 tx=0x7f8964005960 comp rx=0 tx=0).stop 2024-03-24T09:46:54.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.393+0000 7f8970205700 1 -- 172.21.15.22:0/3907438941 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8968100bd0 msgr2=0x7f8968196ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:54.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.393+0000 7f8970205700 1 --2- 172.21.15.22:0/3907438941 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8968100bd0 0x7f8968196ac0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f895800c8f0 tx=0x7f895800cc00 comp rx=0 tx=0).stop 2024-03-24T09:46:54.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.394+0000 7f8970205700 1 -- 172.21.15.22:0/3907438941 shutdown_connections 2024-03-24T09:46:54.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.394+0000 7f8970205700 1 --2- 172.21.15.22:0/3907438941 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f895406cce0 0x7f895406f1a0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:54.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.394+0000 7f8970205700 1 --2- 172.21.15.22:0/3907438941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f89680ff9d0 0x7f8968196580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:54.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.394+0000 7f8970205700 1 --2- 172.21.15.22:0/3907438941 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8968100bd0 0x7f8968196ac0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:54.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.394+0000 7f8970205700 1 -- 172.21.15.22:0/3907438941 >> 172.21.15.22:0/3907438941 conn(0x7f89680faf70 msgr2=0x7f8968068530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:54.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.394+0000 7f8970205700 1 -- 172.21.15.22:0/3907438941 shutdown_connections 2024-03-24T09:46:54.394 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:54.394+0000 7f8970205700 1 -- 172.21.15.22:0/3907438941 wait complete. 2024-03-24T09:46:54.404 INFO:teuthology.orchestra.run.smithi022.stdout:true 2024-03-24T09:46:54.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:54 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:54.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:54 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:46:54.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:54 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/3907438941' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-03-24T09:46:54.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:54 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:54.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:54 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:46:54.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:54 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/3907438941' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-03-24T09:46:55.108 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-03-24T09:46:55.121 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T09:46:55.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.237+0000 7fbcfd470640 1 -- 172.21.15.22:0/2173585749 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbcf80ffff0 msgr2=0x7fbcf8100410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:55.241 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.237+0000 7fbcfd470640 1 --2- 172.21.15.22:0/2173585749 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbcf80ffff0 0x7fbcf8100410 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fbce8009880 tx=0x7fbce802bce0 comp rx=0 tx=0).stop 2024-03-24T09:46:55.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.237+0000 7fbcfd470640 1 -- 172.21.15.22:0/2173585749 shutdown_connections 2024-03-24T09:46:55.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.237+0000 7fbcfd470640 1 --2- 172.21.15.22:0/2173585749 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcf81011e0 0x7fbcf8101660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:55.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.237+0000 7fbcfd470640 1 --2- 172.21.15.22:0/2173585749 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbcf80ffff0 0x7fbcf8100410 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:55.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.237+0000 7fbcfd470640 1 -- 172.21.15.22:0/2173585749 >> 172.21.15.22:0/2173585749 conn(0x7fbcf80fb560 msgr2=0x7fbcf80fd9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:55.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.237+0000 7fbcfd470640 1 -- 172.21.15.22:0/2173585749 shutdown_connections 2024-03-24T09:46:55.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.237+0000 7fbcfd470640 1 -- 172.21.15.22:0/2173585749 wait complete. 2024-03-24T09:46:55.242 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.238+0000 7fbcfd470640 1 Processor -- start 2024-03-24T09:46:55.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.238+0000 7fbcfd470640 1 -- start start 2024-03-24T09:46:55.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.238+0000 7fbcfd470640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcf80ffff0 0x7fbcf8074020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:55.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.238+0000 7fbcfd470640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbcf81011e0 0x7fbcf80726b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:55.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.238+0000 7fbcfd470640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf8072bf0 con 0x7fbcf81011e0 2024-03-24T09:46:55.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.238+0000 7fbcfd470640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf8072d60 con 0x7fbcf80ffff0 2024-03-24T09:46:55.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.238+0000 7fbceffff640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbcf81011e0 0x7fbcf80726b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:55.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.238+0000 7fbceffff640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbcf81011e0 0x7fbcf80726b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:45184/0 (socket says 172.21.15.22:45184) 2024-03-24T09:46:55.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.238+0000 7fbceffff640 1 -- 172.21.15.22:0/1937975942 learned_addr learned my addr 172.21.15.22:0/1937975942 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:46:55.243 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.238+0000 7fbcf6ffd640 1 --2- 172.21.15.22:0/1937975942 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcf80ffff0 0x7fbcf8074020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:55.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.238+0000 7fbceffff640 1 -- 172.21.15.22:0/1937975942 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcf80ffff0 msgr2=0x7fbcf8074020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:55.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.238+0000 7fbceffff640 1 --2- 172.21.15.22:0/1937975942 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcf80ffff0 0x7fbcf8074020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:55.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.238+0000 7fbceffff640 1 -- 172.21.15.22:0/1937975942 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbce8009560 con 0x7fbcf81011e0 2024-03-24T09:46:55.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.239+0000 7fbceffff640 1 --2- 172.21.15.22:0/1937975942 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbcf81011e0 0x7fbcf80726b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fbce000d6f0 tx=0x7fbce000dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:55.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.239+0000 7fbcf4ff9640 1 -- 172.21.15.22:0/1937975942 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbce0009a30 con 0x7fbcf81011e0 2024-03-24T09:46:55.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.239+0000 7fbcf4ff9640 1 -- 172.21.15.22:0/1937975942 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbce00054e0 con 0x7fbcf81011e0 2024-03-24T09:46:55.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.239+0000 7fbcfd470640 1 -- 172.21.15.22:0/1937975942 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcf8073040 con 0x7fbcf81011e0 2024-03-24T09:46:55.244 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.239+0000 7fbcf4ff9640 1 -- 172.21.15.22:0/1937975942 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbce0010460 con 0x7fbcf81011e0 2024-03-24T09:46:55.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.239+0000 7fbcfd470640 1 -- 172.21.15.22:0/1937975942 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcf819db80 con 0x7fbcf81011e0 2024-03-24T09:46:55.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.240+0000 7fbcfd470640 1 -- 172.21.15.22:0/1937975942 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcf804e030 con 0x7fbcf81011e0 2024-03-24T09:46:55.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.240+0000 7fbcf4ff9640 1 -- 172.21.15.22:0/1937975942 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7fbce0009b90 con 0x7fbcf81011e0 2024-03-24T09:46:55.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.241+0000 7fbcf4ff9640 1 --2- 172.21.15.22:0/1937975942 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fbce406cc50 0x7fbce406f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:55.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.241+0000 7fbcf6ffd640 1 --2- 172.21.15.22:0/1937975942 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fbce406cc50 0x7fbce406f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:55.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.241+0000 7fbcf4ff9640 1 -- 172.21.15.22:0/1937975942 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fbce008be50 con 0x7fbcf81011e0 2024-03-24T09:46:55.245 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.241+0000 7fbcf6ffd640 1 --2- 172.21.15.22:0/1937975942 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fbce406cc50 0x7fbce406f110 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbce8006010 tx=0x7fbce802c920 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:55.246 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.245+0000 7fbcf4ff9640 1 -- 172.21.15.22:0/1937975942 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbce00561c0 con 0x7fbcf81011e0 2024-03-24T09:46:55.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.429+0000 7fbcfd470640 1 -- 172.21.15.22:0/1937975942 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbcf8108bf0 con 0x7fbcf81011e0 2024-03-24T09:46:55.430 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.430+0000 7fbcf4ff9640 1 -- 172.21.15.22:0/1937975942 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+15151 (secure 0 0 0) 0x7fbce00597e0 con 0x7fbcf81011e0 2024-03-24T09:46:55.431 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:46:55.431 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":46,"fsid":"45356448-e9c2-11ee-95cd-87774f69a715","created":"2024-03-24T09:39:25.502234+0000","modified":"2024-03-24T09:46:45.149139+0000","last_up_change":"2024-03-24T09:45:12.567478+0000","last_in_change":"2024-03-24T09:44:53.111067+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T09:43:56.788035+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-03-24T09:46:28.601117+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-03-24T09:46:28.908013+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ba411044-c790-44b5-881e-4e86675a868b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":3431970421},{"type":"v1","addr":"172.21.15.22:6803","nonce":3431970421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":3431970421},{"type":"v1","addr":"172.21.15.22:6805","nonce":3431970421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":3431970421},{"type":"v1","addr":"172.21.15.22:6809","nonce":3431970421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":3431970421},{"type":"v1","addr":"172.21.15.22:6807","nonce":3431970421}]},"public_addr":"172.21.15.22:6803/3431970421","cluster_addr":"172.21.15.22:6805/3431970421","heartbeat_back_addr":"172.21.15.22:6809/3431970421","heartbeat_front_addr":"172.21.15.22:6807/3431970421","state":["exists","up"]},{"osd":1,"uuid":"2bc9ee9b-4d76-4df5-bc42-92f243ba6c9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":1675089147},{"type":"v1","addr":"172.21.15.22:6811","nonce":1675089147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":1675089147},{"type":"v1","addr":"172.21.15.22:6813","nonce":1675089147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":1675089147},{"type":"v1","addr":"172.21.15.22:6817","nonce":1675089147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":1675089147},{"type":"v1","addr":"172.21.15.22:6815","nonce":1675089147}]},"public_addr":"172.21.15.22:6811/1675089147","cluster_addr":"172.21.15.22:6813/1675089147","heartbeat_back_addr":"172.21.15.22:6817/1675089147","heartbeat_front_addr":"172.21.15.22:6815/1675089147","state":["exists","up"]},{"osd":2,"uuid":"1e0c3afd-b47b-4de8-ab3c-e25770557c6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":1538987315},{"type":"v1","addr":"172.21.15.22:6819","nonce":1538987315}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":1538987315},{"type":"v1","addr":"172.21.15.22:6821","nonce":1538987315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":1538987315},{"type":"v1","addr":"172.21.15.22:6825","nonce":1538987315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":1538987315},{"type":"v1","addr":"172.21.15.22:6823","nonce":1538987315}]},"public_addr":"172.21.15.22:6819/1538987315","cluster_addr":"172.21.15.22:6821/1538987315","heartbeat_back_addr":"172.21.15.22:6825/1538987315","heartbeat_front_addr":"172.21.15.22:6823/1538987315","state":["exists","up"]},{"osd":3,"uuid":"56f51449-b4b8-4874-9469-497ce984a5f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":2405415706},{"type":"v1","addr":"172.21.15.155:6801","nonce":2405415706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":2405415706},{"type":"v1","addr":"172.21.15.155:6803","nonce":2405415706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":2405415706},{"type":"v1","addr":"172.21.15.155:6807","nonce":2405415706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":2405415706},{"type":"v1","addr":"172.21.15.155:6805","nonce":2405415706}]},"public_addr":"172.21.15.155:6801/2405415706","cluster_addr":"172.21.15.155:6803/2405415706","heartbeat_back_addr":"172.21.15.155:6807/2405415706","heartbeat_front_addr":"172.21.15.155:6805/2405415706","state":["exists","up"]},{"osd":4,"uuid":"1af86297-1514-495a-8d74-6d0237f82316","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":2965421758},{"type":"v1","addr":"172.21.15.155:6809","nonce":2965421758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":2965421758},{"type":"v1","addr":"172.21.15.155:6811","nonce":2965421758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":2965421758},{"type":"v1","addr":"172.21.15.155:6815","nonce":2965421758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":2965421758},{"type":"v1","addr":"172.21.15.155:6813","nonce":2965421758}]},"public_addr":"172.21.15.155:6809/2965421758","cluster_addr":"172.21.15.155:6811/2965421758","heartbeat_back_addr":"172.21.15.155:6815/2965421758","heartbeat_front_addr":"172.21.15.155:6813/2965421758","state":["exists","up"]},{"osd":5,"uuid":"aa6a8e29-123e-4b1f-8bed-8b207cfadce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3664800012},{"type":"v1","addr":"172.21.15.155:6817","nonce":3664800012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3664800012},{"type":"v1","addr":"172.21.15.155:6819","nonce":3664800012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3664800012},{"type":"v1","addr":"172.21.15.155:6823","nonce":3664800012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3664800012},{"type":"v1","addr":"172.21.15.155:6821","nonce":3664800012}]},"public_addr":"172.21.15.155:6817/3664800012","cluster_addr":"172.21.15.155:6819/3664800012","heartbeat_back_addr":"172.21.15.155:6823/3664800012","heartbeat_front_addr":"172.21.15.155:6821/3664800012","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:42:57.929167+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:43:25.482421+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:43:54.589274+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:44:18.882506+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:44:44.731273+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:45:10.622995+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.22:6827/3295717728":"2024-03-25T09:46:45.149102+0000","172.21.15.22:0/1186018607":"2024-03-25T09:40:30.945227+0000","172.21.15.22:6800/2423771373":"2024-03-25T09:39:58.019245+0000","172.21.15.22:6800/2979399337":"2024-03-25T09:41:38.451303+0000","172.21.15.22:6801/2423771373":"2024-03-25T09:39:58.019245+0000","172.21.15.22:0/3835020536":"2024-03-25T09:41:38.451303+0000","172.21.15.22:0/2826976179":"2024-03-25T09:39:58.019245+0000","172.21.15.22:0/1182234699":"2024-03-25T09:40:30.945227+0000","172.21.15.22:6800/3036078499":"2024-03-25T09:40:30.945227+0000","172.21.15.22:6826/3295717728":"2024-03-25T09:46:45.149102+0000","172.21.15.22:6801/3036078499":"2024-03-25T09:40:30.945227+0000","172.21.15.22:0/363626647":"2024-03-25T09:39:58.019245+0000","172.21.15.22:0/774594813":"2024-03-25T09:40:30.945227+0000","172.21.15.22:0/1096383264":"2024-03-25T09:41:38.451303+0000","172.21.15.22:6801/2979399337":"2024-03-25T09:41:38.451303+0000","172.21.15.22:0/2928401592":"2024-03-25T09:39:58.019245+0000","172.21.15.22:0/632176108":"2024-03-25T09:41:38.451303+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T09:46:55.433 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.432+0000 7fbcfd470640 1 -- 172.21.15.22:0/1937975942 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fbce406cc50 msgr2=0x7fbce406f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:55.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.432+0000 7fbcfd470640 1 --2- 172.21.15.22:0/1937975942 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fbce406cc50 0x7fbce406f110 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbce8006010 tx=0x7fbce802c920 comp rx=0 tx=0).stop 2024-03-24T09:46:55.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.432+0000 7fbcfd470640 1 -- 172.21.15.22:0/1937975942 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbcf81011e0 msgr2=0x7fbcf80726b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:55.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.432+0000 7fbcfd470640 1 --2- 172.21.15.22:0/1937975942 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbcf81011e0 0x7fbcf80726b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fbce000d6f0 tx=0x7fbce000dbb0 comp rx=0 tx=0).stop 2024-03-24T09:46:55.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.432+0000 7fbcfd470640 1 -- 172.21.15.22:0/1937975942 shutdown_connections 2024-03-24T09:46:55.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.432+0000 7fbcfd470640 1 --2- 172.21.15.22:0/1937975942 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fbce406cc50 0x7fbce406f110 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:55.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.432+0000 7fbcfd470640 1 --2- 172.21.15.22:0/1937975942 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbcf81011e0 0x7fbcf80726b0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:55.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.432+0000 7fbcfd470640 1 --2- 172.21.15.22:0/1937975942 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcf80ffff0 0x7fbcf8074020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:55.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.432+0000 7fbcfd470640 1 -- 172.21.15.22:0/1937975942 >> 172.21.15.22:0/1937975942 conn(0x7fbcf80fb560 msgr2=0x7fbcf8104830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:55.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.433+0000 7fbcfd470640 1 -- 172.21.15.22:0/1937975942 shutdown_connections 2024-03-24T09:46:55.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.433+0000 7fbcfd470640 1 -- 172.21.15.22:0/1937975942 wait complete. 2024-03-24T09:46:55.445 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T09:43:56.788035+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-03-24T09:46:28.601117+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-03-24T09:46:28.908013+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T09:46:55.445 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T09:46:55.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.555+0000 7f593cb5c640 1 -- 172.21.15.22:0/1551181929 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f59381033a0 msgr2=0x7f5938105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:55.556 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.556+0000 7f593cb5c640 1 --2- 172.21.15.22:0/1551181929 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f59381033a0 0x7f5938105790 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f592c0098b0 tx=0x7f592c02bcd0 comp rx=0 tx=0).stop 2024-03-24T09:46:55.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.556+0000 7f593cb5c640 1 -- 172.21.15.22:0/1551181929 shutdown_connections 2024-03-24T09:46:55.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.556+0000 7f593cb5c640 1 --2- 172.21.15.22:0/1551181929 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f59381033a0 0x7f5938105790 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:55.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.556+0000 7f593cb5c640 1 --2- 172.21.15.22:0/1551181929 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5938100a70 0x7f5938102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:55.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.556+0000 7f593cb5c640 1 -- 172.21.15.22:0/1551181929 >> 172.21.15.22:0/1551181929 conn(0x7f59380fa460 msgr2=0x7f59380fc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:55.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.556+0000 7f593cb5c640 1 -- 172.21.15.22:0/1551181929 shutdown_connections 2024-03-24T09:46:55.557 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.556+0000 7f593cb5c640 1 -- 172.21.15.22:0/1551181929 wait complete. 2024-03-24T09:46:55.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.557+0000 7f593cb5c640 1 Processor -- start 2024-03-24T09:46:55.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.557+0000 7f593cb5c640 1 -- start start 2024-03-24T09:46:55.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.557+0000 7f593cb5c640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5938100a70 0x7f5938100390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:55.560 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.557+0000 7f593cb5c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59381033a0 0x7f59380fea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:55.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.557+0000 7f593cb5c640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59380fef60 con 0x7f5938100a70 2024-03-24T09:46:55.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.557+0000 7f593cb5c640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59380ff0d0 con 0x7f59381033a0 2024-03-24T09:46:55.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.557+0000 7f593659c640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5938100a70 0x7f5938100390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:55.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.557+0000 7f593659c640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5938100a70 0x7f5938100390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:45196/0 (socket says 172.21.15.22:45196) 2024-03-24T09:46:55.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.557+0000 7f593659c640 1 -- 172.21.15.22:0/1399543311 learned_addr learned my addr 172.21.15.22:0/1399543311 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:46:55.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.557+0000 7f5935d9b640 1 --2- 172.21.15.22:0/1399543311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59381033a0 0x7f59380fea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:55.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.557+0000 7f593659c640 1 -- 172.21.15.22:0/1399543311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59381033a0 msgr2=0x7f59380fea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:55.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.557+0000 7f593659c640 1 --2- 172.21.15.22:0/1399543311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59381033a0 0x7f59380fea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:55.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.557+0000 7f593659c640 1 -- 172.21.15.22:0/1399543311 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f592c009560 con 0x7f5938100a70 2024-03-24T09:46:55.561 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.558+0000 7f593659c640 1 --2- 172.21.15.22:0/1399543311 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5938100a70 0x7f5938100390 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f592800c8f0 tx=0x7f592800cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:55.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.558+0000 7f59277fe640 1 -- 172.21.15.22:0/1399543311 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5928007b80 con 0x7f5938100a70 2024-03-24T09:46:55.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.558+0000 7f593cb5c640 1 -- 172.21.15.22:0/1399543311 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59380ff3b0 con 0x7f5938100a70 2024-03-24T09:46:55.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.558+0000 7f59277fe640 1 -- 172.21.15.22:0/1399543311 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f59280054e0 con 0x7f5938100a70 2024-03-24T09:46:55.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.558+0000 7f59277fe640 1 -- 172.21.15.22:0/1399543311 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f592800f450 con 0x7f5938100a70 2024-03-24T09:46:55.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.558+0000 7f593cb5c640 1 -- 172.21.15.22:0/1399543311 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f593819db50 con 0x7f5938100a70 2024-03-24T09:46:55.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.559+0000 7f593cb5c640 1 -- 172.21.15.22:0/1399543311 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f593804e030 con 0x7f5938100a70 2024-03-24T09:46:55.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.560+0000 7f59277fe640 1 -- 172.21.15.22:0/1399543311 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f5928004f60 con 0x7f5938100a70 2024-03-24T09:46:55.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.560+0000 7f59277fe640 1 --2- 172.21.15.22:0/1399543311 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f592006cc60 0x7f592006f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:55.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.560+0000 7f59277fe640 1 -- 172.21.15.22:0/1399543311 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f592808c050 con 0x7f5938100a70 2024-03-24T09:46:55.562 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.560+0000 7f5935d9b640 1 --2- 172.21.15.22:0/1399543311 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f592006cc60 0x7f592006f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:55.563 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.561+0000 7f5935d9b640 1 --2- 172.21.15.22:0/1399543311 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f592006cc60 0x7f592006f120 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f592c000c00 tx=0x7f592c005fd0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:55.565 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.565+0000 7f59277fe640 1 -- 172.21.15.22:0/1399543311 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5928056470 con 0x7f5938100a70 2024-03-24T09:46:55.776 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.776+0000 7f593cb5c640 1 -- 172.21.15.22:0/1399543311 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f593819de70 con 0x7f5938100a70 2024-03-24T09:46:55.776 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.776+0000 7f59277fe640 1 -- 172.21.15.22:0/1399543311 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5928059a90 con 0x7f5938100a70 2024-03-24T09:46:55.777 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 1 2024-03-24T09:46:55.779 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.778+0000 7f593cb5c640 1 -- 172.21.15.22:0/1399543311 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f592006cc60 msgr2=0x7f592006f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:55.779 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.778+0000 7f593cb5c640 1 --2- 172.21.15.22:0/1399543311 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f592006cc60 0x7f592006f120 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f592c000c00 tx=0x7f592c005fd0 comp rx=0 tx=0).stop 2024-03-24T09:46:55.779 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.778+0000 7f593cb5c640 1 -- 172.21.15.22:0/1399543311 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5938100a70 msgr2=0x7f5938100390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:55.779 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.778+0000 7f593cb5c640 1 --2- 172.21.15.22:0/1399543311 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5938100a70 0x7f5938100390 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f592800c8f0 tx=0x7f592800cdb0 comp rx=0 tx=0).stop 2024-03-24T09:46:55.779 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.778+0000 7f593cb5c640 1 -- 172.21.15.22:0/1399543311 shutdown_connections 2024-03-24T09:46:55.779 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.778+0000 7f593cb5c640 1 --2- 172.21.15.22:0/1399543311 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f592006cc60 0x7f592006f120 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:55.779 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.778+0000 7f593cb5c640 1 --2- 172.21.15.22:0/1399543311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59381033a0 0x7f59380fea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:55.779 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.778+0000 7f593cb5c640 1 --2- 172.21.15.22:0/1399543311 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5938100a70 0x7f5938100390 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:55.780 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.778+0000 7f593cb5c640 1 -- 172.21.15.22:0/1399543311 >> 172.21.15.22:0/1399543311 conn(0x7f59380fa460 msgr2=0x7f59380fc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:55.780 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.778+0000 7f593cb5c640 1 -- 172.21.15.22:0/1399543311 shutdown_connections 2024-03-24T09:46:55.780 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.778+0000 7f593cb5c640 1 -- 172.21.15.22:0/1399543311 wait complete. 2024-03-24T09:46:55.781 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:55 smithi022 ceph-mon[29512]: pgmap v192: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 9.6 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T09:46:55.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:55 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:55.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:55 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:55.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:55 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/1937975942' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:46:55.789 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-03-24T09:46:55.898 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.895+0000 7fefd49fb640 1 -- 172.21.15.22:0/1124497538 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fefd00fee60 msgr2=0x7fefd00ff280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:55.898 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.895+0000 7fefd49fb640 1 --2- 172.21.15.22:0/1124497538 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fefd00fee60 0x7fefd00ff280 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fefc0009880 tx=0x7fefc002bce0 comp rx=0 tx=0).stop 2024-03-24T09:46:55.899 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.895+0000 7fefd49fb640 1 -- 172.21.15.22:0/1124497538 shutdown_connections 2024-03-24T09:46:55.899 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.895+0000 7fefd49fb640 1 --2- 172.21.15.22:0/1124497538 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fefd0100040 0x7fefd01004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:55.899 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.895+0000 7fefd49fb640 1 --2- 172.21.15.22:0/1124497538 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fefd00fee60 0x7fefd00ff280 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:55.899 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.895+0000 7fefd49fb640 1 -- 172.21.15.22:0/1124497538 >> 172.21.15.22:0/1124497538 conn(0x7fefd00fa400 msgr2=0x7fefd00fc860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:55.899 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.895+0000 7fefd49fb640 1 -- 172.21.15.22:0/1124497538 shutdown_connections 2024-03-24T09:46:55.899 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.895+0000 7fefd49fb640 1 -- 172.21.15.22:0/1124497538 wait complete. 2024-03-24T09:46:55.899 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.895+0000 7fefd49fb640 1 Processor -- start 2024-03-24T09:46:55.899 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.895+0000 7fefd49fb640 1 -- start start 2024-03-24T09:46:55.899 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.896+0000 7fefd49fb640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fefd00fee60 0x7fefd0194b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:55.899 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.896+0000 7fefd49fb640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fefd0100040 0x7fefd0195070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:55.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.896+0000 7fefd49fb640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd0195640 con 0x7fefd00fee60 2024-03-24T09:46:55.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.896+0000 7fefd49fb640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd01957b0 con 0x7fefd0100040 2024-03-24T09:46:55.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.896+0000 7fefce59c640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fefd00fee60 0x7fefd0194b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:55.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.896+0000 7fefce59c640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fefd00fee60 0x7fefd0194b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:45208/0 (socket says 172.21.15.22:45208) 2024-03-24T09:46:55.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.896+0000 7fefce59c640 1 -- 172.21.15.22:0/567533858 learned_addr learned my addr 172.21.15.22:0/567533858 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:46:55.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.896+0000 7fefcdd9b640 1 --2- 172.21.15.22:0/567533858 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fefd0100040 0x7fefd0195070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:55.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.896+0000 7fefce59c640 1 -- 172.21.15.22:0/567533858 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fefd0100040 msgr2=0x7fefd0195070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:55.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.896+0000 7fefce59c640 1 --2- 172.21.15.22:0/567533858 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fefd0100040 0x7fefd0195070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:55.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.896+0000 7fefce59c640 1 -- 172.21.15.22:0/567533858 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefc0009560 con 0x7fefd00fee60 2024-03-24T09:46:55.900 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.896+0000 7fefcdd9b640 1 --2- 172.21.15.22:0/567533858 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fefd0100040 0x7fefd0195070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T09:46:55.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.896+0000 7fefce59c640 1 --2- 172.21.15.22:0/567533858 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fefd00fee60 0x7fefd0194b30 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fefc0000c00 tx=0x7fefc0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:55.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.896+0000 7fefbf7fe640 1 -- 172.21.15.22:0/567533858 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefc00037c0 con 0x7fefd00fee60 2024-03-24T09:46:55.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.896+0000 7fefd49fb640 1 -- 172.21.15.22:0/567533858 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefd019a190 con 0x7fefd00fee60 2024-03-24T09:46:55.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.896+0000 7fefbf7fe640 1 -- 172.21.15.22:0/567533858 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fefc0003de0 con 0x7fefd00fee60 2024-03-24T09:46:55.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.896+0000 7fefbf7fe640 1 -- 172.21.15.22:0/567533858 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefc0034440 con 0x7fefd00fee60 2024-03-24T09:46:55.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.896+0000 7fefd49fb640 1 -- 172.21.15.22:0/567533858 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefd019a710 con 0x7fefd00fee60 2024-03-24T09:46:55.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.897+0000 7fefd49fb640 1 -- 172.21.15.22:0/567533858 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefd0107a70 con 0x7fefd00fee60 2024-03-24T09:46:55.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.898+0000 7fefbf7fe640 1 -- 172.21.15.22:0/567533858 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7fefc0005040 con 0x7fefd00fee60 2024-03-24T09:46:55.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.898+0000 7fefbf7fe640 1 --2- 172.21.15.22:0/567533858 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fefb806cc50 0x7fefb806f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:55.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.898+0000 7fefbf7fe640 1 -- 172.21.15.22:0/567533858 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fefc00ae7c0 con 0x7fefd00fee60 2024-03-24T09:46:55.901 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.898+0000 7fefcdd9b640 1 --2- 172.21.15.22:0/567533858 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fefb806cc50 0x7fefb806f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:55.902 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.899+0000 7fefcdd9b640 1 --2- 172.21.15.22:0/567533858 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fefb806cc50 0x7fefb806f110 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fefc4006fd0 tx=0x7fefc4008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:55.903 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:55.903+0000 7fefbf7fe640 1 -- 172.21.15.22:0/567533858 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fefc0078bf0 con 0x7fefd00fee60 2024-03-24T09:46:55.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:55 smithi155 ceph-mon[37016]: pgmap v192: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 9.6 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T09:46:55.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:55 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:55.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:55 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:46:55.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:55 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/1937975942' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:46:56.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.091+0000 7fefd49fb640 1 -- 172.21.15.22:0/567533858 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7fefd0107c00 con 0x7fefd00fee60 2024-03-24T09:46:56.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.091+0000 7fefbf7fe640 1 -- 172.21.15.22:0/567533858 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7fefc007c210 con 0x7fefd00fee60 2024-03-24T09:46:56.091 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 32 2024-03-24T09:46:56.093 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.093+0000 7fefd49fb640 1 -- 172.21.15.22:0/567533858 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fefb806cc50 msgr2=0x7fefb806f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:56.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.093+0000 7fefd49fb640 1 --2- 172.21.15.22:0/567533858 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fefb806cc50 0x7fefb806f110 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fefc4006fd0 tx=0x7fefc4008040 comp rx=0 tx=0).stop 2024-03-24T09:46:56.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.093+0000 7fefd49fb640 1 -- 172.21.15.22:0/567533858 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fefd00fee60 msgr2=0x7fefd0194b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:56.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.093+0000 7fefd49fb640 1 --2- 172.21.15.22:0/567533858 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fefd00fee60 0x7fefd0194b30 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fefc0000c00 tx=0x7fefc0004270 comp rx=0 tx=0).stop 2024-03-24T09:46:56.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.093+0000 7fefd49fb640 1 -- 172.21.15.22:0/567533858 shutdown_connections 2024-03-24T09:46:56.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.093+0000 7fefd49fb640 1 --2- 172.21.15.22:0/567533858 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fefb806cc50 0x7fefb806f110 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:56.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.093+0000 7fefd49fb640 1 --2- 172.21.15.22:0/567533858 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fefd0100040 0x7fefd0195070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:56.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.093+0000 7fefd49fb640 1 --2- 172.21.15.22:0/567533858 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fefd00fee60 0x7fefd0194b30 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:56.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.093+0000 7fefd49fb640 1 -- 172.21.15.22:0/567533858 >> 172.21.15.22:0/567533858 conn(0x7fefd00fa400 msgr2=0x7fefd0103360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:56.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.093+0000 7fefd49fb640 1 -- 172.21.15.22:0/567533858 shutdown_connections 2024-03-24T09:46:56.094 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.093+0000 7fefd49fb640 1 -- 172.21.15.22:0/567533858 wait complete. 2024-03-24T09:46:56.104 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-03-24T09:46:56.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.210+0000 7fbba08ae640 1 -- 172.21.15.22:0/466320453 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbb9c1011b0 msgr2=0x7fbb9c101630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:56.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.210+0000 7fbba08ae640 1 --2- 172.21.15.22:0/466320453 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbb9c1011b0 0x7fbb9c101630 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fbb84009880 tx=0x7fbb8402bce0 comp rx=0 tx=0).stop 2024-03-24T09:46:56.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.210+0000 7fbba08ae640 1 -- 172.21.15.22:0/466320453 shutdown_connections 2024-03-24T09:46:56.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.210+0000 7fbba08ae640 1 --2- 172.21.15.22:0/466320453 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbb9c1011b0 0x7fbb9c101630 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:56.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.210+0000 7fbba08ae640 1 --2- 172.21.15.22:0/466320453 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb9c0fffc0 0x7fbb9c1003e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:56.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.210+0000 7fbba08ae640 1 -- 172.21.15.22:0/466320453 >> 172.21.15.22:0/466320453 conn(0x7fbb9c0fb530 msgr2=0x7fbb9c0fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:56.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.211+0000 7fbba08ae640 1 -- 172.21.15.22:0/466320453 shutdown_connections 2024-03-24T09:46:56.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.211+0000 7fbba08ae640 1 -- 172.21.15.22:0/466320453 wait complete. 2024-03-24T09:46:56.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.211+0000 7fbba08ae640 1 Processor -- start 2024-03-24T09:46:56.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.211+0000 7fbba08ae640 1 -- start start 2024-03-24T09:46:56.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.211+0000 7fbba08ae640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbb9c0fffc0 0x7fbb9c195b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:56.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.211+0000 7fbba08ae640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb9c1011b0 0x7fbb9c196040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:56.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.211+0000 7fbba08ae640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9c196610 con 0x7fbb9c0fffc0 2024-03-24T09:46:56.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.211+0000 7fbba08ae640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9c196780 con 0x7fbb9c1011b0 2024-03-24T09:46:56.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.211+0000 7fbb9a59c640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbb9c0fffc0 0x7fbb9c195b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:56.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.212+0000 7fbb9a59c640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbb9c0fffc0 0x7fbb9c195b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:45226/0 (socket says 172.21.15.22:45226) 2024-03-24T09:46:56.214 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.212+0000 7fbb9a59c640 1 -- 172.21.15.22:0/3391189424 learned_addr learned my addr 172.21.15.22:0/3391189424 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:46:56.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.212+0000 7fbb99d9b640 1 --2- 172.21.15.22:0/3391189424 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb9c1011b0 0x7fbb9c196040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:56.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.212+0000 7fbb9a59c640 1 -- 172.21.15.22:0/3391189424 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb9c1011b0 msgr2=0x7fbb9c196040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:56.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.212+0000 7fbb9a59c640 1 --2- 172.21.15.22:0/3391189424 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb9c1011b0 0x7fbb9c196040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:56.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.212+0000 7fbb9a59c640 1 -- 172.21.15.22:0/3391189424 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb84009560 con 0x7fbb9c0fffc0 2024-03-24T09:46:56.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.212+0000 7fbb99d9b640 1 --2- 172.21.15.22:0/3391189424 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb9c1011b0 0x7fbb9c196040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:46:56.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.212+0000 7fbb9a59c640 1 --2- 172.21.15.22:0/3391189424 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbb9c0fffc0 0x7fbb9c195b00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fbb9400d6f0 tx=0x7fbb9400dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:56.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.212+0000 7fbb937fe640 1 -- 172.21.15.22:0/3391189424 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb94009a30 con 0x7fbb9c0fffc0 2024-03-24T09:46:56.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.212+0000 7fbb937fe640 1 -- 172.21.15.22:0/3391189424 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbb94009b90 con 0x7fbb9c0fffc0 2024-03-24T09:46:56.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.212+0000 7fbba08ae640 1 -- 172.21.15.22:0/3391189424 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb9c19b1c0 con 0x7fbb9c0fffc0 2024-03-24T09:46:56.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.212+0000 7fbb937fe640 1 -- 172.21.15.22:0/3391189424 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb94003e70 con 0x7fbb9c0fffc0 2024-03-24T09:46:56.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.212+0000 7fbba08ae640 1 -- 172.21.15.22:0/3391189424 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb9c19b7a0 con 0x7fbb9c0fffc0 2024-03-24T09:46:56.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.213+0000 7fbba08ae640 1 -- 172.21.15.22:0/3391189424 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb9c18fb30 con 0x7fbb9c0fffc0 2024-03-24T09:46:56.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.214+0000 7fbb937fe640 1 -- 172.21.15.22:0/3391189424 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7fbb94009d00 con 0x7fbb9c0fffc0 2024-03-24T09:46:56.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.214+0000 7fbb937fe640 1 --2- 172.21.15.22:0/3391189424 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fbb8806cc50 0x7fbb8806f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:56.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.214+0000 7fbb937fe640 1 -- 172.21.15.22:0/3391189424 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fbb9408bc70 con 0x7fbb9c0fffc0 2024-03-24T09:46:56.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.214+0000 7fbb99d9b640 1 --2- 172.21.15.22:0/3391189424 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fbb8806cc50 0x7fbb8806f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:56.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.215+0000 7fbb99d9b640 1 --2- 172.21.15.22:0/3391189424 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fbb8806cc50 0x7fbb8806f110 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c196fc0 tx=0x7fbb8402c680 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:56.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.219+0000 7fbb937fe640 1 -- 172.21.15.22:0/3391189424 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbb94056060 con 0x7fbb9c0fffc0 2024-03-24T09:46:56.402 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.402+0000 7fbba08ae640 1 -- 172.21.15.22:0/3391189424 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fbb9c108ba0 con 0x7fbb9c0fffc0 2024-03-24T09:46:56.402 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.402+0000 7fbb937fe640 1 -- 172.21.15.22:0/3391189424 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7fbb940200b0 con 0x7fbb9c0fffc0 2024-03-24T09:46:56.402 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 32 2024-03-24T09:46:56.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.404+0000 7fbba08ae640 1 -- 172.21.15.22:0/3391189424 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fbb8806cc50 msgr2=0x7fbb8806f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:56.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.404+0000 7fbba08ae640 1 --2- 172.21.15.22:0/3391189424 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fbb8806cc50 0x7fbb8806f110 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c196fc0 tx=0x7fbb8402c680 comp rx=0 tx=0).stop 2024-03-24T09:46:56.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.404+0000 7fbba08ae640 1 -- 172.21.15.22:0/3391189424 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbb9c0fffc0 msgr2=0x7fbb9c195b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:56.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.404+0000 7fbba08ae640 1 --2- 172.21.15.22:0/3391189424 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbb9c0fffc0 0x7fbb9c195b00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fbb9400d6f0 tx=0x7fbb9400dbb0 comp rx=0 tx=0).stop 2024-03-24T09:46:56.405 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.404+0000 7fbba08ae640 1 -- 172.21.15.22:0/3391189424 shutdown_connections 2024-03-24T09:46:56.405 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.404+0000 7fbba08ae640 1 --2- 172.21.15.22:0/3391189424 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fbb8806cc50 0x7fbb8806f110 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:56.405 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.404+0000 7fbba08ae640 1 --2- 172.21.15.22:0/3391189424 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb9c1011b0 0x7fbb9c196040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:56.405 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.404+0000 7fbba08ae640 1 --2- 172.21.15.22:0/3391189424 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbb9c0fffc0 0x7fbb9c195b00 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:56.405 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.404+0000 7fbba08ae640 1 -- 172.21.15.22:0/3391189424 >> 172.21.15.22:0/3391189424 conn(0x7fbb9c0fb530 msgr2=0x7fbb9c1047b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:56.405 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.404+0000 7fbba08ae640 1 -- 172.21.15.22:0/3391189424 shutdown_connections 2024-03-24T09:46:56.405 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.404+0000 7fbba08ae640 1 -- 172.21.15.22:0/3391189424 wait complete. 2024-03-24T09:46:56.415 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:46:56.531 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.530+0000 7fd18d1d2640 1 -- 172.21.15.22:0/3835170445 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd1881033e0 msgr2=0x7fd1881057d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:56.531 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.530+0000 7fd18d1d2640 1 --2- 172.21.15.22:0/3835170445 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd1881033e0 0x7fd1881057d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd17c009880 tx=0x7fd17c02bce0 comp rx=0 tx=0).stop 2024-03-24T09:46:56.531 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.530+0000 7fd18d1d2640 1 -- 172.21.15.22:0/3835170445 shutdown_connections 2024-03-24T09:46:56.531 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.530+0000 7fd18d1d2640 1 --2- 172.21.15.22:0/3835170445 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd1881033e0 0x7fd1881057d0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:56.531 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.530+0000 7fd18d1d2640 1 --2- 172.21.15.22:0/3835170445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd188100ab0 0x7fd188102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:56.531 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.530+0000 7fd18d1d2640 1 -- 172.21.15.22:0/3835170445 >> 172.21.15.22:0/3835170445 conn(0x7fd1880fa4a0 msgr2=0x7fd1880fc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:56.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.530+0000 7fd18d1d2640 1 -- 172.21.15.22:0/3835170445 shutdown_connections 2024-03-24T09:46:56.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.530+0000 7fd18d1d2640 1 -- 172.21.15.22:0/3835170445 wait complete. 2024-03-24T09:46:56.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.531+0000 7fd18d1d2640 1 Processor -- start 2024-03-24T09:46:56.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.531+0000 7fd18d1d2640 1 -- start start 2024-03-24T09:46:56.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.531+0000 7fd18d1d2640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd188100ab0 0x7fd188195a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:56.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.531+0000 7fd18d1d2640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd1881033e0 0x7fd188195fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:56.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.531+0000 7fd18d1d2640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1881965a0 con 0x7fd1881033e0 2024-03-24T09:46:56.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.531+0000 7fd18d1d2640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd188196710 con 0x7fd188100ab0 2024-03-24T09:46:56.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.532+0000 7fd18659c640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd1881033e0 0x7fd188195fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:56.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.532+0000 7fd18659c640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd1881033e0 0x7fd188195fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:45250/0 (socket says 172.21.15.22:45250) 2024-03-24T09:46:56.533 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.532+0000 7fd18659c640 1 -- 172.21.15.22:0/3221822457 learned_addr learned my addr 172.21.15.22:0/3221822457 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:46:56.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.532+0000 7fd186d9d640 1 --2- 172.21.15.22:0/3221822457 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd188100ab0 0x7fd188195a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:56.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.532+0000 7fd18659c640 1 -- 172.21.15.22:0/3221822457 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd188100ab0 msgr2=0x7fd188195a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:56.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.532+0000 7fd18659c640 1 --2- 172.21.15.22:0/3221822457 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd188100ab0 0x7fd188195a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:56.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.532+0000 7fd18659c640 1 -- 172.21.15.22:0/3221822457 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17c009560 con 0x7fd1881033e0 2024-03-24T09:46:56.534 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.532+0000 7fd18659c640 1 --2- 172.21.15.22:0/3221822457 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd1881033e0 0x7fd188195fd0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd17c009230 tx=0x7fd17c02c750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:56.535 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.534+0000 7fd16ffff640 1 -- 172.21.15.22:0/3221822457 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd17c0035f0 con 0x7fd1881033e0 2024-03-24T09:46:56.535 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.534+0000 7fd18d1d2640 1 -- 172.21.15.22:0/3221822457 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd18819b0f0 con 0x7fd1881033e0 2024-03-24T09:46:56.535 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.534+0000 7fd16ffff640 1 -- 172.21.15.22:0/3221822457 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd17c003c10 con 0x7fd1881033e0 2024-03-24T09:46:56.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.534+0000 7fd16ffff640 1 -- 172.21.15.22:0/3221822457 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd17c034440 con 0x7fd1881033e0 2024-03-24T09:46:56.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.534+0000 7fd18d1d2640 1 -- 172.21.15.22:0/3221822457 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd18819b670 con 0x7fd1881033e0 2024-03-24T09:46:56.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.535+0000 7fd18d1d2640 1 -- 172.21.15.22:0/3221822457 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd188108b60 con 0x7fd1881033e0 2024-03-24T09:46:56.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.535+0000 7fd16ffff640 1 -- 172.21.15.22:0/3221822457 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7fd17c034630 con 0x7fd1881033e0 2024-03-24T09:46:56.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.535+0000 7fd16ffff640 1 --2- 172.21.15.22:0/3221822457 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fd17006cc60 0x7fd17006f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:46:56.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.535+0000 7fd16ffff640 1 -- 172.21.15.22:0/3221822457 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fd17c0ae5e0 con 0x7fd1881033e0 2024-03-24T09:46:56.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.535+0000 7fd186d9d640 1 --2- 172.21.15.22:0/3221822457 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fd17006cc60 0x7fd17006f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:46:56.536 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.536+0000 7fd186d9d640 1 --2- 172.21.15.22:0/3221822457 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fd17006cc60 0x7fd17006f120 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd178009bd0 tx=0x7fd178005f90 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:46:56.540 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.540+0000 7fd16ffff640 1 -- 172.21.15.22:0/3221822457 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd17c078a70 con 0x7fd1881033e0 2024-03-24T09:46:56.696 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:56 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/1399543311' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:46:56.696 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:56 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/567533858' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-24T09:46:56.696 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:56 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/3391189424' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-24T09:46:56.760 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.760+0000 7fd18d1d2640 1 -- 172.21.15.22:0/3221822457 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd188108cf0 con 0x7fd1881033e0 2024-03-24T09:46:56.760 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.760+0000 7fd16ffff640 1 -- 172.21.15.22:0/3221822457 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4839 (secure 0 0 0) 0x7fd17c07c090 con 0x7fd1881033e0 2024-03-24T09:46:56.761 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-24T09:46:56.761 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14504,"name":"cephfs.smithi022.mhcbrm","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6827/3956813571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":3956813571},{"type":"v1","addr":"172.21.15.22:6827","nonce":3956813571}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24275,"name":"cephfs.smithi155.dakojr","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.155:6827/3192784756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":3192784756},{"type":"v1","addr":"172.21.15.155:6827","nonce":3192784756}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":15,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T09:46:29.665102+0000","modified":"2024-03-24T09:46:48.168028+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24269},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14496":{"gid":14496,"name":"cephfs.smithi022.qbmomh","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.22:6829/2909541441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":2909541441},{"type":"v1","addr":"172.21.15.22:6829","nonce":2909541441}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24269":{"gid":24269,"name":"cephfs.smithi155.pwhqcw","rank":0,"incarnation":11,"state":"up:active","state_seq":6,"addr":"172.21.15.155:6825/4090729184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":4090729184},{"type":"v1","addr":"172.21.15.155:6825","nonce":4090729184}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-24T09:46:56.761 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 15 2024-03-24T09:46:56.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.762+0000 7fd18d1d2640 1 -- 172.21.15.22:0/3221822457 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fd17006cc60 msgr2=0x7fd17006f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:56.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.762+0000 7fd18d1d2640 1 --2- 172.21.15.22:0/3221822457 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fd17006cc60 0x7fd17006f120 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd178009bd0 tx=0x7fd178005f90 comp rx=0 tx=0).stop 2024-03-24T09:46:56.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.763+0000 7fd18d1d2640 1 -- 172.21.15.22:0/3221822457 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd1881033e0 msgr2=0x7fd188195fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:46:56.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.763+0000 7fd18d1d2640 1 --2- 172.21.15.22:0/3221822457 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd1881033e0 0x7fd188195fd0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd17c009230 tx=0x7fd17c02c750 comp rx=0 tx=0).stop 2024-03-24T09:46:56.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.763+0000 7fd18d1d2640 1 -- 172.21.15.22:0/3221822457 shutdown_connections 2024-03-24T09:46:56.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.763+0000 7fd18d1d2640 1 --2- 172.21.15.22:0/3221822457 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fd17006cc60 0x7fd17006f120 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:56.763 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.763+0000 7fd18d1d2640 1 --2- 172.21.15.22:0/3221822457 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd1881033e0 0x7fd188195fd0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:56.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.763+0000 7fd18d1d2640 1 --2- 172.21.15.22:0/3221822457 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd188100ab0 0x7fd188195a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:46:56.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.763+0000 7fd18d1d2640 1 -- 172.21.15.22:0/3221822457 >> 172.21.15.22:0/3221822457 conn(0x7fd1880fa4a0 msgr2=0x7fd1880fc890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:46:56.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.763+0000 7fd18d1d2640 1 -- 172.21.15.22:0/3221822457 shutdown_connections 2024-03-24T09:46:56.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:56.763+0000 7fd18d1d2640 1 -- 172.21.15.22:0/3221822457 wait complete. 2024-03-24T09:46:56.776 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 15, 'max_mds': 1, 'flags': 50} 2024-03-24T09:46:56.776 INFO:teuthology.run_tasks:Running task kclient... 2024-03-24T09:46:56.800 INFO:tasks.kclient:Mounting kernel clients... 2024-03-24T09:46:56.800 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2024-03-24T09:46:56.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:46:56.800 DEBUG:teuthology.orchestra.run.smithi022:> ip netns list 2024-03-24T09:46:56.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:46:56.816 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip link delete ceph-brx 2024-03-24T09:46:56.884 INFO:teuthology.orchestra.run.smithi022.stderr:Cannot find device "ceph-brx" 2024-03-24T09:46:56.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T09:46:56.886 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:46:56.886 DEBUG:teuthology.orchestra.run.smithi155:> ip netns list 2024-03-24T09:46:56.906 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:46:56.906 DEBUG:teuthology.orchestra.run.smithi155:> sudo ip link delete ceph-brx 2024-03-24T09:46:56.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:56 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/1399543311' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:46:56.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:56 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/567533858' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-24T09:46:56.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:56 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/3391189424' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-24T09:46:56.983 INFO:teuthology.orchestra.run.smithi155.stderr:Cannot find device "ceph-brx" 2024-03-24T09:46:56.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T09:46:56.984 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-03-24T09:46:56.984 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-24T09:46:56.985 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T09:46:56.985 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph fs ls 2024-03-24T09:46:57.223 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:46:57.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:57 smithi022 ceph-mon[29512]: pgmap v193: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 8.7 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-24T09:46:57.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:57 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/3221822457' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:46:57.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:57 smithi155 ceph-mon[37016]: pgmap v193: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 8.7 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-24T09:46:57.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:57 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/3221822457' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:46:59.756 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:46:59 smithi022 ceph-mon[29512]: pgmap v194: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 7.4 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T09:46:59.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:46:59 smithi155 ceph-mon[37016]: pgmap v194: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 7.4 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T09:47:00.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:59.997+0000 7fa2a780f700 1 -- 172.21.15.22:0/1184994495 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa2a0106a70 msgr2=0x7fa2a0106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:00.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:59.997+0000 7fa2a780f700 1 --2- 172.21.15.22:0/1184994495 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa2a0106a70 0x7fa2a0106e50 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa298009910 tx=0x7fa298009c20 comp rx=0 tx=0).stop 2024-03-24T09:47:00.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:59.997+0000 7fa2a780f700 1 -- 172.21.15.22:0/1184994495 shutdown_connections 2024-03-24T09:47:00.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:59.997+0000 7fa2a780f700 1 --2- 172.21.15.22:0/1184994495 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2a0100a70 0x7fa2a0100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:00.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:59.997+0000 7fa2a780f700 1 --2- 172.21.15.22:0/1184994495 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa2a0106a70 0x7fa2a0106e50 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:00.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:59.997+0000 7fa2a780f700 1 -- 172.21.15.22:0/1184994495 >> 172.21.15.22:0/1184994495 conn(0x7fa2a00fc5a0 msgr2=0x7fa2a00fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:47:00.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:59.997+0000 7fa2a780f700 1 -- 172.21.15.22:0/1184994495 shutdown_connections 2024-03-24T09:47:00.001 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:59.997+0000 7fa2a780f700 1 -- 172.21.15.22:0/1184994495 wait complete. 2024-03-24T09:47:00.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:59.998+0000 7fa2a780f700 1 Processor -- start 2024-03-24T09:47:00.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:59.998+0000 7fa2a780f700 1 -- start start 2024-03-24T09:47:00.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:59.998+0000 7fa2a780f700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa2a0100a70 0x7fa2a0073070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:47:00.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:59.998+0000 7fa2a780f700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2a0106a70 0x7fa2a00735b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:47:00.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:59.998+0000 7fa2a780f700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a0073af0 con 0x7fa2a0100a70 2024-03-24T09:47:00.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:59.998+0000 7fa2a780f700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a0073c30 con 0x7fa2a0106a70 2024-03-24T09:47:00.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:59.998+0000 7fa2a55ab700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa2a0100a70 0x7fa2a0073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:47:00.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:59.998+0000 7fa2a55ab700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa2a0100a70 0x7fa2a0073070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:33426/0 (socket says 172.21.15.22:33426) 2024-03-24T09:47:00.002 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:59.998+0000 7fa2a55ab700 1 -- 172.21.15.22:0/3382134657 learned_addr learned my addr 172.21.15.22:0/3382134657 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:47:00.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:59.998+0000 7fa2a4daa700 1 --2- 172.21.15.22:0/3382134657 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2a0106a70 0x7fa2a00735b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:47:00.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:59.998+0000 7fa2a55ab700 1 -- 172.21.15.22:0/3382134657 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2a0106a70 msgr2=0x7fa2a00735b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:00.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:59.998+0000 7fa2a55ab700 1 --2- 172.21.15.22:0/3382134657 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2a0106a70 0x7fa2a00735b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:00.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:59.998+0000 7fa2a55ab700 1 -- 172.21.15.22:0/3382134657 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2980095f0 con 0x7fa2a0100a70 2024-03-24T09:47:00.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:59.998+0000 7fa2a4daa700 1 --2- 172.21.15.22:0/3382134657 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2a0106a70 0x7fa2a00735b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T09:47:00.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:59.998+0000 7fa2a55ab700 1 --2- 172.21.15.22:0/3382134657 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa2a0100a70 0x7fa2a0073070 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa298006010 tx=0x7fa29800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:47:00.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:59.999+0000 7fa29e7fc700 1 -- 172.21.15.22:0/3382134657 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa29801d070 con 0x7fa2a0100a70 2024-03-24T09:47:00.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:59.999+0000 7fa29e7fc700 1 -- 172.21.15.22:0/3382134657 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa29800f460 con 0x7fa2a0100a70 2024-03-24T09:47:00.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:59.999+0000 7fa2a780f700 1 -- 172.21.15.22:0/3382134657 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2a0073eb0 con 0x7fa2a0100a70 2024-03-24T09:47:00.003 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:59.999+0000 7fa29e7fc700 1 -- 172.21.15.22:0/3382134657 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa298017630 con 0x7fa2a0100a70 2024-03-24T09:47:00.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:46:59.999+0000 7fa2a780f700 1 -- 172.21.15.22:0/3382134657 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2a01a2f30 con 0x7fa2a0100a70 2024-03-24T09:47:00.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:00.000+0000 7fa2a780f700 1 -- 172.21.15.22:0/3382134657 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2a0109070 con 0x7fa2a0100a70 2024-03-24T09:47:00.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:00.000+0000 7fa29e7fc700 1 -- 172.21.15.22:0/3382134657 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7fa29800f5d0 con 0x7fa2a0100a70 2024-03-24T09:47:00.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:00.001+0000 7fa29e7fc700 1 --2- 172.21.15.22:0/3382134657 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fa28806ccf0 0x7fa28806f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:47:00.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:00.001+0000 7fa29e7fc700 1 -- 172.21.15.22:0/3382134657 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fa29808d6f0 con 0x7fa2a0100a70 2024-03-24T09:47:00.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:00.001+0000 7fa2a4daa700 1 --2- 172.21.15.22:0/3382134657 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fa28806ccf0 0x7fa28806f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:47:00.004 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:00.001+0000 7fa2a4daa700 1 --2- 172.21.15.22:0/3382134657 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fa28806ccf0 0x7fa28806f1b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa2a00747a0 tx=0x7fa294008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:47:00.005 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:00.005+0000 7fa29e7fc700 1 -- 172.21.15.22:0/3382134657 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa298057ba0 con 0x7fa2a0100a70 2024-03-24T09:47:00.263 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:00.263+0000 7fa2a780f700 1 -- 172.21.15.22:0/3382134657 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fa2a0108a50 con 0x7fa2a0100a70 2024-03-24T09:47:00.263 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:00.263+0000 7fa29e7fc700 1 -- 172.21.15.22:0/3382134657 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) v1 ==== 53+0+83 (secure 0 0 0) 0x7fa29805b1c0 con 0x7fa2a0100a70 2024-03-24T09:47:00.264 INFO:teuthology.orchestra.run.smithi022.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-03-24T09:47:00.265 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:00.265+0000 7fa2a780f700 1 -- 172.21.15.22:0/3382134657 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fa28806ccf0 msgr2=0x7fa28806f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:00.265 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:00.265+0000 7fa2a780f700 1 --2- 172.21.15.22:0/3382134657 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fa28806ccf0 0x7fa28806f1b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa2a00747a0 tx=0x7fa294008040 comp rx=0 tx=0).stop 2024-03-24T09:47:00.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:00.265+0000 7fa2a780f700 1 -- 172.21.15.22:0/3382134657 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa2a0100a70 msgr2=0x7fa2a0073070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:00.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:00.265+0000 7fa2a780f700 1 --2- 172.21.15.22:0/3382134657 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa2a0100a70 0x7fa2a0073070 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa298006010 tx=0x7fa29800be20 comp rx=0 tx=0).stop 2024-03-24T09:47:00.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:00.265+0000 7fa2a780f700 1 -- 172.21.15.22:0/3382134657 shutdown_connections 2024-03-24T09:47:00.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:00.265+0000 7fa2a780f700 1 --2- 172.21.15.22:0/3382134657 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fa28806ccf0 0x7fa28806f1b0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:00.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:00.265+0000 7fa2a780f700 1 --2- 172.21.15.22:0/3382134657 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa2a0100a70 0x7fa2a0073070 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:00.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:00.265+0000 7fa2a780f700 1 --2- 172.21.15.22:0/3382134657 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2a0106a70 0x7fa2a00735b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:00.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:00.265+0000 7fa2a780f700 1 -- 172.21.15.22:0/3382134657 >> 172.21.15.22:0/3382134657 conn(0x7fa2a00fc5a0 msgr2=0x7fa2a00fd2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:47:00.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:00.265+0000 7fa2a780f700 1 -- 172.21.15.22:0/3382134657 shutdown_connections 2024-03-24T09:47:00.266 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:00.265+0000 7fa2a780f700 1 -- 172.21.15.22:0/3382134657 wait complete. 2024-03-24T09:47:00.850 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T09:47:00.850 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T09:47:00.850 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi022.front.sepia.ceph.com 2024-03-24T09:47:00.851 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T09:47:00.851 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T09:47:00.851 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T09:47:00.851 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T09:47:00.851 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T09:47:00.851 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T09:47:00.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:47:00.851 DEBUG:teuthology.orchestra.run.smithi022:> ip addr 2024-03-24T09:47:00.868 INFO:teuthology.orchestra.run.smithi022.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T09:47:00.868 INFO:teuthology.orchestra.run.smithi022.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T09:47:00.868 INFO:teuthology.orchestra.run.smithi022.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T09:47:00.868 INFO:teuthology.orchestra.run.smithi022.stdout: valid_lft forever preferred_lft forever 2024-03-24T09:47:00.868 INFO:teuthology.orchestra.run.smithi022.stdout: inet6 ::1/128 scope host 2024-03-24T09:47:00.869 INFO:teuthology.orchestra.run.smithi022.stdout: valid_lft forever preferred_lft forever 2024-03-24T09:47:00.869 INFO:teuthology.orchestra.run.smithi022.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T09:47:00.869 INFO:teuthology.orchestra.run.smithi022.stdout: link/ether 0c:c4:7a:6b:fd:9a brd ff:ff:ff:ff:ff:ff 2024-03-24T09:47:00.869 INFO:teuthology.orchestra.run.smithi022.stdout: altname enp7s0f0 2024-03-24T09:47:00.869 INFO:teuthology.orchestra.run.smithi022.stdout:3: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T09:47:00.869 INFO:teuthology.orchestra.run.smithi022.stdout: link/ether 0c:c4:7a:59:5d:e4 brd ff:ff:ff:ff:ff:ff 2024-03-24T09:47:00.869 INFO:teuthology.orchestra.run.smithi022.stdout: altname enp1s0f0 2024-03-24T09:47:00.869 INFO:teuthology.orchestra.run.smithi022.stdout: inet 172.21.15.22/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-24T09:47:00.869 INFO:teuthology.orchestra.run.smithi022.stdout: valid_lft 42182sec preferred_lft 42182sec 2024-03-24T09:47:00.869 INFO:teuthology.orchestra.run.smithi022.stdout: inet6 fe80::ec4:7aff:fe59:5de4/64 scope link noprefixroute 2024-03-24T09:47:00.870 INFO:teuthology.orchestra.run.smithi022.stdout: valid_lft forever preferred_lft forever 2024-03-24T09:47:00.870 INFO:teuthology.orchestra.run.smithi022.stdout:4: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T09:47:00.870 INFO:teuthology.orchestra.run.smithi022.stdout: link/ether 0c:c4:7a:6b:fd:9b brd ff:ff:ff:ff:ff:ff 2024-03-24T09:47:00.870 INFO:teuthology.orchestra.run.smithi022.stdout: altname enp7s0f1 2024-03-24T09:47:00.870 INFO:teuthology.orchestra.run.smithi022.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T09:47:00.870 INFO:teuthology.orchestra.run.smithi022.stdout: link/ether 0c:c4:7a:59:5d:e5 brd ff:ff:ff:ff:ff:ff 2024-03-24T09:47:00.870 INFO:teuthology.orchestra.run.smithi022.stdout: altname enp1s0f1 2024-03-24T09:47:00.871 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-24T09:47:00.871 DEBUG:teuthology.orchestra.run.smithi022:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T09:47:00.871 DEBUG:teuthology.orchestra.run.smithi022:> set -e 2024-03-24T09:47:00.871 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip link add name ceph-brx type bridge 2024-03-24T09:47:00.871 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip addr flush dev ceph-brx 2024-03-24T09:47:00.871 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip link set ceph-brx up 2024-03-24T09:47:00.871 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-24T09:47:00.871 DEBUG:teuthology.orchestra.run.smithi022:> ') 2024-03-24T09:47:00.958 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T09:47:01.091 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T09:47:01.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:47:01.098 DEBUG:teuthology.orchestra.run.smithi022:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T09:47:01.125 INFO:teuthology.orchestra.run.smithi022.stdout:1 2024-03-24T09:47:01.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:47:01.127 DEBUG:teuthology.orchestra.run.smithi022:> route 2024-03-24T09:47:01.201 INFO:teuthology.orchestra.run.smithi022.stdout:Kernel IP routing table 2024-03-24T09:47:01.201 INFO:teuthology.orchestra.run.smithi022.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T09:47:01.201 INFO:teuthology.orchestra.run.smithi022.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-24T09:47:01.201 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-24T09:47:01.201 INFO:teuthology.orchestra.run.smithi022.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T09:47:01.202 DEBUG:teuthology.orchestra.run.smithi022:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T09:47:01.202 DEBUG:teuthology.orchestra.run.smithi022:> set -e 2024-03-24T09:47:01.202 DEBUG:teuthology.orchestra.run.smithi022:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-24T09:47:01.202 DEBUG:teuthology.orchestra.run.smithi022:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-24T09:47:01.202 DEBUG:teuthology.orchestra.run.smithi022:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-24T09:47:01.203 DEBUG:teuthology.orchestra.run.smithi022:> ') 2024-03-24T09:47:01.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:01 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:47:01.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:01 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:47:01.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:01 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:47:01.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:01 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:47:01.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:01 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:47:01.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:01 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/3382134657' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-24T09:47:01.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:01 smithi022 ceph-mon[29512]: pgmap v195: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 6.6 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T09:47:01.288 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T09:47:01.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:01 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:47:01.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:01 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:47:01.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:01 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:47:01.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:01 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:47:01.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:01 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:47:01.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:01 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/3382134657' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-24T09:47:01.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:01 smithi155 ceph-mon[37016]: pgmap v195: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 6.6 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T09:47:01.486 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T09:47:01.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:47:01.492 DEBUG:teuthology.orchestra.run.smithi022:> ip netns list 2024-03-24T09:47:01.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:47:01.548 DEBUG:teuthology.orchestra.run.smithi022:> ip netns list-id 2024-03-24T09:47:01.602 DEBUG:teuthology.orchestra.run.smithi022:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T09:47:01.602 DEBUG:teuthology.orchestra.run.smithi022:> set -e 2024-03-24T09:47:01.602 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T09:47:01.602 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-24T09:47:01.602 DEBUG:teuthology.orchestra.run.smithi022:> ') 2024-03-24T09:47:01.691 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T09:47:01.719 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T09:47:01.725 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-24T09:47:01.726 DEBUG:teuthology.orchestra.run.smithi022:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T09:47:01.726 DEBUG:teuthology.orchestra.run.smithi022:> set -e 2024-03-24T09:47:01.726 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-24T09:47:01.726 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-24T09:47:01.726 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-24T09:47:01.726 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-24T09:47:01.726 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-24T09:47:01.726 DEBUG:teuthology.orchestra.run.smithi022:> ') 2024-03-24T09:47:01.773 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T09:47:01.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T09:47:01.867 DEBUG:teuthology.orchestra.run.smithi022:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T09:47:01.867 DEBUG:teuthology.orchestra.run.smithi022:> set -e 2024-03-24T09:47:01.867 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip link set brx.0 up 2024-03-24T09:47:01.867 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip link set dev brx.0 master ceph-brx 2024-03-24T09:47:01.867 DEBUG:teuthology.orchestra.run.smithi022:> ') 2024-03-24T09:47:01.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T09:47:01.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T09:47:01.948 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T09:47:01.948 DEBUG:teuthology.orchestra.run.smithi022:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T09:47:01.962 INFO:teuthology.orchestra.run.smithi022.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T09:47:01.963 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T09:47:01.963 DEBUG:teuthology.orchestra.run.smithi022:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T09:47:02.018 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-24T09:47:02.018 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:47:02.018 DEBUG:teuthology.orchestra.run.smithi022:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T09:47:02.113 INFO:teuthology.orchestra.run.smithi022.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T09:47:02.113 INFO:teuthology.orchestra.run.smithi022.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T09:47:02.113 INFO:teuthology.orchestra.run.smithi022.stdout:invalid new device string format 2024-03-24T09:47:02.231 INFO:teuthology.orchestra.run.smithi022.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T09:47:02.232 INFO:teuthology.orchestra.run.smithi022.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T09:47:02.232 INFO:teuthology.orchestra.run.smithi022.stdout:invalid new device string format 2024-03-24T09:47:02.232 INFO:teuthology.orchestra.run.smithi022.stdout:mount.ceph: resolved to: "172.21.15.22:3300,172.21.15.155:3300" 2024-03-24T09:47:02.232 INFO:teuthology.orchestra.run.smithi022.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T09:47:02.237 INFO:teuthology.orchestra.run.smithi022.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T09:47:02.237 INFO:teuthology.orchestra.run.smithi022.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T09:47:02.237 INFO:teuthology.orchestra.run.smithi022.stdout:invalid new device string format 2024-03-24T09:47:02.237 INFO:teuthology.orchestra.run.smithi022.stdout:mount.ceph: resolved to: "172.21.15.22:3300,172.21.15.155:3300" 2024-03-24T09:47:02.238 INFO:teuthology.orchestra.run.smithi022.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T09:47:02.238 INFO:teuthology.orchestra.run.smithi022.stdout:mount.ceph: resolved to: "172.21.15.22:6789,172.21.15.155:6789" 2024-03-24T09:47:02.238 INFO:teuthology.orchestra.run.smithi022.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.22:6789,172.21.15.155:6789:/ 2024-03-24T09:47:02.238 INFO:teuthology.orchestra.run.smithi022.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=45356448-e9c2-11ee-95cd-87774f69a715" will pass to kernel 2024-03-24T09:47:02.238 INFO:teuthology.orchestra.run.smithi022.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-24T09:47:02.238 INFO:teuthology.orchestra.run.smithi022.stdout: You just mounted a file system that supports labels which does not 2024-03-24T09:47:02.239 INFO:teuthology.orchestra.run.smithi022.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-24T09:47:02.239 INFO:teuthology.orchestra.run.smithi022.stdout: applications will generate AVC messages and not be allowed access to 2024-03-24T09:47:02.239 INFO:teuthology.orchestra.run.smithi022.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-24T09:47:02.240 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-24T09:47:02.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:47:02.241 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T09:47:02.269 DEBUG:teuthology.orchestra.run.smithi022:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T09:47:02.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:02 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-24T09:47:02.367 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T09:47:02.373 INFO:teuthology.orchestra.run.smithi022.stdout:client24303 2024-03-24T09:47:02.373 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T09:47:02.373 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T09:47:02.390 DEBUG:teuthology.orchestra.run.smithi022:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T09:47:02.482 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:02 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-24T09:47:02.488 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T09:47:02.494 INFO:teuthology.orchestra.run.smithi022.stdout:client24303 2024-03-24T09:47:02.494 DEBUG:teuthology.orchestra.run.smithi022:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T09:47:02.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:02 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-24T09:47:02.552 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T09:47:02.557 INFO:teuthology.orchestra.run.smithi022.stdout:client24303 2024-03-24T09:47:02.558 DEBUG:teuthology.orchestra.run.smithi022:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/45356448-e9c2-11ee-95cd-87774f69a715.client24303/status') 2024-03-24T09:47:02.608 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T09:47:02.624 INFO:teuthology.orchestra.run.smithi022.stdout:instance: client.24303 (3)172.21.15.22:0/1561644150 2024-03-24T09:47:02.624 INFO:teuthology.orchestra.run.smithi022.stdout:blocklisted: false 2024-03-24T09:47:02.624 INFO:teuthology.orchestra.run.smithi022.stderr:0+1 records in 2024-03-24T09:47:02.624 INFO:teuthology.orchestra.run.smithi022.stderr:0+1 records out 2024-03-24T09:47:02.625 INFO:teuthology.orchestra.run.smithi022.stderr:71 bytes copied, 2.0666e-05 s, 3.4 MB/s 2024-03-24T09:47:02.625 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T09:47:02.631 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-03-24T09:47:02.631 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-24T09:47:02.632 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-24T09:47:02.632 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -- ceph fs ls 2024-03-24T09:47:02.844 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:47:03.836 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:03 smithi022 ceph-mon[29512]: pgmap v196: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 6.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T09:47:03.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:03 smithi155 ceph-mon[37016]: pgmap v196: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 6.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T09:47:04.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.400+0000 7fd786ac1700 1 -- 172.21.15.22:0/3994704851 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd780101c10 msgr2=0x7fd780101ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:04.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.400+0000 7fd786ac1700 1 --2- 172.21.15.22:0/3994704851 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd780101c10 0x7fd780101ff0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd774009910 tx=0x7fd774009c20 comp rx=0 tx=0).stop 2024-03-24T09:47:04.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.400+0000 7fd786ac1700 1 -- 172.21.15.22:0/3994704851 shutdown_connections 2024-03-24T09:47:04.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.400+0000 7fd786ac1700 1 --2- 172.21.15.22:0/3994704851 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd780102530 0x7fd78010aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:04.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.400+0000 7fd786ac1700 1 --2- 172.21.15.22:0/3994704851 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd780101c10 0x7fd780101ff0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:04.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.400+0000 7fd786ac1700 1 -- 172.21.15.22:0/3994704851 >> 172.21.15.22:0/3994704851 conn(0x7fd780076640 msgr2=0x7fd7800fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:47:04.403 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.400+0000 7fd786ac1700 1 -- 172.21.15.22:0/3994704851 shutdown_connections 2024-03-24T09:47:04.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.400+0000 7fd786ac1700 1 -- 172.21.15.22:0/3994704851 wait complete. 2024-03-24T09:47:04.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.401+0000 7fd786ac1700 1 Processor -- start 2024-03-24T09:47:04.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.401+0000 7fd786ac1700 1 -- start start 2024-03-24T09:47:04.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.401+0000 7fd786ac1700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd780101c10 0x7fd78019c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:47:04.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.401+0000 7fd786ac1700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd780102530 0x7fd78019cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:47:04.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.401+0000 7fd786ac1700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7801968b0 con 0x7fd780101c10 2024-03-24T09:47:04.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.401+0000 7fd786ac1700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd780196a20 con 0x7fd780102530 2024-03-24T09:47:04.404 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.401+0000 7fd78485d700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd780101c10 0x7fd78019c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:47:04.405 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.401+0000 7fd78485d700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd780101c10 0x7fd78019c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:33450/0 (socket says 172.21.15.22:33450) 2024-03-24T09:47:04.405 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.401+0000 7fd78485d700 1 -- 172.21.15.22:0/3655572635 learned_addr learned my addr 172.21.15.22:0/3655572635 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:47:04.405 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.401+0000 7fd77ffff700 1 --2- 172.21.15.22:0/3655572635 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd780102530 0x7fd78019cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:47:04.405 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.402+0000 7fd78485d700 1 -- 172.21.15.22:0/3655572635 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd780102530 msgr2=0x7fd78019cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:04.405 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.402+0000 7fd78485d700 1 --2- 172.21.15.22:0/3655572635 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd780102530 0x7fd78019cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:04.405 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.402+0000 7fd78485d700 1 -- 172.21.15.22:0/3655572635 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7740095f0 con 0x7fd780101c10 2024-03-24T09:47:04.405 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.402+0000 7fd77ffff700 1 --2- 172.21.15.22:0/3655572635 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd780102530 0x7fd78019cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:47:04.405 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.402+0000 7fd78485d700 1 --2- 172.21.15.22:0/3655572635 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd780101c10 0x7fd78019c800 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd774006010 tx=0x7fd77400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:47:04.405 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.402+0000 7fd77dffb700 1 -- 172.21.15.22:0/3655572635 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd77401d070 con 0x7fd780101c10 2024-03-24T09:47:04.405 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.402+0000 7fd77dffb700 1 -- 172.21.15.22:0/3655572635 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd77400f460 con 0x7fd780101c10 2024-03-24T09:47:04.405 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.402+0000 7fd77dffb700 1 -- 172.21.15.22:0/3655572635 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd774017630 con 0x7fd780101c10 2024-03-24T09:47:04.406 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.402+0000 7fd786ac1700 1 -- 172.21.15.22:0/3655572635 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd780196ca0 con 0x7fd780101c10 2024-03-24T09:47:04.406 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.402+0000 7fd786ac1700 1 -- 172.21.15.22:0/3655572635 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7801970f0 con 0x7fd780101c10 2024-03-24T09:47:04.407 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.403+0000 7fd786ac1700 1 -- 172.21.15.22:0/3655572635 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd780068d60 con 0x7fd780101c10 2024-03-24T09:47:04.407 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.404+0000 7fd77dffb700 1 -- 172.21.15.22:0/3655572635 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7fd77400f5d0 con 0x7fd780101c10 2024-03-24T09:47:04.407 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.404+0000 7fd77dffb700 1 --2- 172.21.15.22:0/3655572635 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fd77006cce0 0x7fd77006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:47:04.407 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.404+0000 7fd77dffb700 1 -- 172.21.15.22:0/3655572635 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fd77408d7e0 con 0x7fd780101c10 2024-03-24T09:47:04.407 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.404+0000 7fd77ffff700 1 --2- 172.21.15.22:0/3655572635 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fd77006cce0 0x7fd77006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:47:04.407 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.405+0000 7fd77ffff700 1 --2- 172.21.15.22:0/3655572635 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fd77006cce0 0x7fd77006f1a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd780198020 tx=0x7fd76c008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:47:04.409 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.410+0000 7fd77dffb700 1 -- 172.21.15.22:0/3655572635 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd774057bf0 con 0x7fd780101c10 2024-03-24T09:47:04.655 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.654+0000 7fd786ac1700 1 -- 172.21.15.22:0/3655572635 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd7801051e0 con 0x7fd780101c10 2024-03-24T09:47:04.655 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.655+0000 7fd77dffb700 1 -- 172.21.15.22:0/3655572635 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) v1 ==== 53+0+83 (secure 0 0 0) 0x7fd77405b210 con 0x7fd780101c10 2024-03-24T09:47:04.655 INFO:teuthology.orchestra.run.smithi022.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-03-24T09:47:04.657 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.658+0000 7fd786ac1700 1 -- 172.21.15.22:0/3655572635 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fd77006cce0 msgr2=0x7fd77006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:04.658 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.658+0000 7fd786ac1700 1 --2- 172.21.15.22:0/3655572635 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fd77006cce0 0x7fd77006f1a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd780198020 tx=0x7fd76c008040 comp rx=0 tx=0).stop 2024-03-24T09:47:04.658 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.658+0000 7fd786ac1700 1 -- 172.21.15.22:0/3655572635 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd780101c10 msgr2=0x7fd78019c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:04.658 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.658+0000 7fd786ac1700 1 --2- 172.21.15.22:0/3655572635 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd780101c10 0x7fd78019c800 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd774006010 tx=0x7fd77400be20 comp rx=0 tx=0).stop 2024-03-24T09:47:04.658 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.658+0000 7fd786ac1700 1 -- 172.21.15.22:0/3655572635 shutdown_connections 2024-03-24T09:47:04.658 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.658+0000 7fd786ac1700 1 --2- 172.21.15.22:0/3655572635 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fd77006cce0 0x7fd77006f1a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:04.658 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.658+0000 7fd786ac1700 1 --2- 172.21.15.22:0/3655572635 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd780101c10 0x7fd78019c800 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:04.658 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.658+0000 7fd786ac1700 1 --2- 172.21.15.22:0/3655572635 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd780102530 0x7fd78019cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:04.658 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.658+0000 7fd786ac1700 1 -- 172.21.15.22:0/3655572635 >> 172.21.15.22:0/3655572635 conn(0x7fd780076640 msgr2=0x7fd780102f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:47:04.658 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.658+0000 7fd786ac1700 1 -- 172.21.15.22:0/3655572635 shutdown_connections 2024-03-24T09:47:04.659 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:04.658+0000 7fd786ac1700 1 -- 172.21.15.22:0/3655572635 wait complete. 2024-03-24T09:47:05.306 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T09:47:05.307 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T09:47:05.307 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi155.front.sepia.ceph.com 2024-03-24T09:47:05.307 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-24T09:47:05.307 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-24T09:47:05.307 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T09:47:05.307 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T09:47:05.307 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T09:47:05.307 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-24T09:47:05.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:47:05.308 DEBUG:teuthology.orchestra.run.smithi155:> ip addr 2024-03-24T09:47:05.327 INFO:teuthology.orchestra.run.smithi155.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T09:47:05.327 INFO:teuthology.orchestra.run.smithi155.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T09:47:05.327 INFO:teuthology.orchestra.run.smithi155.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T09:47:05.328 INFO:teuthology.orchestra.run.smithi155.stdout: valid_lft forever preferred_lft forever 2024-03-24T09:47:05.328 INFO:teuthology.orchestra.run.smithi155.stdout: inet6 ::1/128 scope host 2024-03-24T09:47:05.328 INFO:teuthology.orchestra.run.smithi155.stdout: valid_lft forever preferred_lft forever 2024-03-24T09:47:05.328 INFO:teuthology.orchestra.run.smithi155.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T09:47:05.328 INFO:teuthology.orchestra.run.smithi155.stdout: link/ether 0c:c4:7a:d9:a0:fa brd ff:ff:ff:ff:ff:ff 2024-03-24T09:47:05.328 INFO:teuthology.orchestra.run.smithi155.stdout: altname enp7s0f0 2024-03-24T09:47:05.328 INFO:teuthology.orchestra.run.smithi155.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T09:47:05.328 INFO:teuthology.orchestra.run.smithi155.stdout: link/ether 0c:c4:7a:d9:a0:fb brd ff:ff:ff:ff:ff:ff 2024-03-24T09:47:05.328 INFO:teuthology.orchestra.run.smithi155.stdout: altname enp7s0f1 2024-03-24T09:47:05.328 INFO:teuthology.orchestra.run.smithi155.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T09:47:05.329 INFO:teuthology.orchestra.run.smithi155.stdout: link/ether 0c:c4:7a:88:6d:ec brd ff:ff:ff:ff:ff:ff 2024-03-24T09:47:05.329 INFO:teuthology.orchestra.run.smithi155.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T09:47:05.329 INFO:teuthology.orchestra.run.smithi155.stdout: link/ether 0c:c4:7a:88:6d:ed brd ff:ff:ff:ff:ff:ff 2024-03-24T09:47:05.329 INFO:teuthology.orchestra.run.smithi155.stdout: inet 172.21.15.155/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T09:47:05.329 INFO:teuthology.orchestra.run.smithi155.stdout: valid_lft 42251sec preferred_lft 42251sec 2024-03-24T09:47:05.329 INFO:teuthology.orchestra.run.smithi155.stdout: inet6 fe80::ec4:7aff:fe88:6ded/64 scope link 2024-03-24T09:47:05.329 INFO:teuthology.orchestra.run.smithi155.stdout: valid_lft forever preferred_lft forever 2024-03-24T09:47:05.329 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-24T09:47:05.330 DEBUG:teuthology.orchestra.run.smithi155:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T09:47:05.330 DEBUG:teuthology.orchestra.run.smithi155:> set -e 2024-03-24T09:47:05.330 DEBUG:teuthology.orchestra.run.smithi155:> sudo ip link add name ceph-brx type bridge 2024-03-24T09:47:05.330 DEBUG:teuthology.orchestra.run.smithi155:> sudo ip addr flush dev ceph-brx 2024-03-24T09:47:05.330 DEBUG:teuthology.orchestra.run.smithi155:> sudo ip link set ceph-brx up 2024-03-24T09:47:05.330 DEBUG:teuthology.orchestra.run.smithi155:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-24T09:47:05.330 DEBUG:teuthology.orchestra.run.smithi155:> ') 2024-03-24T09:47:05.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:47:05 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T09:47:05.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:47:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T09:47:05.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:47:05.536 DEBUG:teuthology.orchestra.run.smithi155:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T09:47:05.564 INFO:teuthology.orchestra.run.smithi155.stdout:1 2024-03-24T09:47:05.566 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:47:05.566 DEBUG:teuthology.orchestra.run.smithi155:> route 2024-03-24T09:47:05.636 INFO:teuthology.orchestra.run.smithi155.stdout:Kernel IP routing table 2024-03-24T09:47:05.636 INFO:teuthology.orchestra.run.smithi155.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T09:47:05.636 INFO:teuthology.orchestra.run.smithi155.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T09:47:05.636 INFO:teuthology.orchestra.run.smithi155.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T09:47:05.636 INFO:teuthology.orchestra.run.smithi155.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T09:47:05.637 DEBUG:teuthology.orchestra.run.smithi155:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T09:47:05.637 DEBUG:teuthology.orchestra.run.smithi155:> set -e 2024-03-24T09:47:05.637 DEBUG:teuthology.orchestra.run.smithi155:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T09:47:05.637 DEBUG:teuthology.orchestra.run.smithi155:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T09:47:05.637 DEBUG:teuthology.orchestra.run.smithi155:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T09:47:05.637 DEBUG:teuthology.orchestra.run.smithi155:> ') 2024-03-24T09:47:05.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:05 smithi155 ceph-mon[37016]: pgmap v197: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 3.9 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-24T09:47:05.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:05 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/3655572635' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-24T09:47:05.725 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:47:05 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T09:47:05.916 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:47:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T09:47:05.922 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:47:05.922 DEBUG:teuthology.orchestra.run.smithi155:> ip netns list 2024-03-24T09:47:05.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:47:05.978 DEBUG:teuthology.orchestra.run.smithi155:> ip netns list-id 2024-03-24T09:47:06.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:05 smithi022 ceph-mon[29512]: pgmap v197: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 3.9 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-24T09:47:06.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:05 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/3655572635' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-24T09:47:06.033 DEBUG:teuthology.orchestra.run.smithi155:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T09:47:06.033 DEBUG:teuthology.orchestra.run.smithi155:> set -e 2024-03-24T09:47:06.033 DEBUG:teuthology.orchestra.run.smithi155:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-24T09:47:06.033 DEBUG:teuthology.orchestra.run.smithi155:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-24T09:47:06.033 DEBUG:teuthology.orchestra.run.smithi155:> ') 2024-03-24T09:47:06.120 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:47:06 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T09:47:06.153 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:47:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T09:47:06.159 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-24T09:47:06.160 DEBUG:teuthology.orchestra.run.smithi155:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T09:47:06.160 DEBUG:teuthology.orchestra.run.smithi155:> set -e 2024-03-24T09:47:06.160 DEBUG:teuthology.orchestra.run.smithi155:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-24T09:47:06.160 DEBUG:teuthology.orchestra.run.smithi155:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-24T09:47:06.160 DEBUG:teuthology.orchestra.run.smithi155:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-24T09:47:06.160 DEBUG:teuthology.orchestra.run.smithi155:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-24T09:47:06.160 DEBUG:teuthology.orchestra.run.smithi155:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-24T09:47:06.160 DEBUG:teuthology.orchestra.run.smithi155:> ') 2024-03-24T09:47:06.209 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:47:06 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T09:47:06.293 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:47:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T09:47:06.299 DEBUG:teuthology.orchestra.run.smithi155:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T09:47:06.299 DEBUG:teuthology.orchestra.run.smithi155:> set -e 2024-03-24T09:47:06.299 DEBUG:teuthology.orchestra.run.smithi155:> sudo ip link set brx.0 up 2024-03-24T09:47:06.299 DEBUG:teuthology.orchestra.run.smithi155:> sudo ip link set dev brx.0 master ceph-brx 2024-03-24T09:47:06.299 DEBUG:teuthology.orchestra.run.smithi155:> ') 2024-03-24T09:47:06.347 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:47:06 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T09:47:06.375 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:47:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T09:47:06.383 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T09:47:06.383 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-24T09:47:06.397 INFO:teuthology.orchestra.run.smithi155.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-24T09:47:06.398 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T09:47:06.398 DEBUG:teuthology.orchestra.run.smithi155:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-24T09:47:06.453 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-24T09:47:06.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:47:06.453 DEBUG:teuthology.orchestra.run.smithi155:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-24T09:47:06.565 INFO:teuthology.orchestra.run.smithi155.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-24T09:47:06.566 INFO:teuthology.orchestra.run.smithi155.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-24T09:47:06.566 INFO:teuthology.orchestra.run.smithi155.stdout:invalid new device string format 2024-03-24T09:47:06.672 INFO:teuthology.orchestra.run.smithi155.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-24T09:47:06.672 INFO:teuthology.orchestra.run.smithi155.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-24T09:47:06.672 INFO:teuthology.orchestra.run.smithi155.stdout:invalid new device string format 2024-03-24T09:47:06.673 INFO:teuthology.orchestra.run.smithi155.stdout:mount.ceph: resolved to: "172.21.15.22:3300,172.21.15.155:3300" 2024-03-24T09:47:06.673 INFO:teuthology.orchestra.run.smithi155.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T09:47:06.678 INFO:teuthology.orchestra.run.smithi155.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-24T09:47:06.678 INFO:teuthology.orchestra.run.smithi155.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-24T09:47:06.678 INFO:teuthology.orchestra.run.smithi155.stdout:invalid new device string format 2024-03-24T09:47:06.678 INFO:teuthology.orchestra.run.smithi155.stdout:mount.ceph: resolved to: "172.21.15.22:3300,172.21.15.155:3300" 2024-03-24T09:47:06.678 INFO:teuthology.orchestra.run.smithi155.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T09:47:06.678 INFO:teuthology.orchestra.run.smithi155.stdout:mount.ceph: resolved to: "172.21.15.22:6789,172.21.15.155:6789" 2024-03-24T09:47:06.678 INFO:teuthology.orchestra.run.smithi155.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.22:6789,172.21.15.155:6789:/ 2024-03-24T09:47:06.678 INFO:teuthology.orchestra.run.smithi155.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=45356448-e9c2-11ee-95cd-87774f69a715" will pass to kernel 2024-03-24T09:47:06.679 INFO:teuthology.orchestra.run.smithi155.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-03-24T09:47:06.679 INFO:teuthology.orchestra.run.smithi155.stdout: You just mounted a file system that supports labels which does not 2024-03-24T09:47:06.679 INFO:teuthology.orchestra.run.smithi155.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-24T09:47:06.679 INFO:teuthology.orchestra.run.smithi155.stdout: applications will generate AVC messages and not be allowed access to 2024-03-24T09:47:06.679 INFO:teuthology.orchestra.run.smithi155.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-24T09:47:06.681 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-24T09:47:06.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:47:06.681 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-24T09:47:06.709 DEBUG:teuthology.orchestra.run.smithi155:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T09:47:06.798 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:47:06 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-24T09:47:06.800 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:47:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T09:47:06.806 INFO:teuthology.orchestra.run.smithi155.stdout:client14546 2024-03-24T09:47:06.806 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-24T09:47:06.807 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T09:47:06.825 DEBUG:teuthology.orchestra.run.smithi155:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T09:47:06.913 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:47:06 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-24T09:47:06.914 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:47:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T09:47:06.920 INFO:teuthology.orchestra.run.smithi155.stdout:client14546 2024-03-24T09:47:06.921 DEBUG:teuthology.orchestra.run.smithi155:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T09:47:06.968 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:47:06 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-24T09:47:06.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:47:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T09:47:06.976 INFO:teuthology.orchestra.run.smithi155.stdout:client14546 2024-03-24T09:47:06.977 DEBUG:teuthology.orchestra.run.smithi155:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/45356448-e9c2-11ee-95cd-87774f69a715.client14546/status') 2024-03-24T09:47:07.025 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:47:07 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T09:47:07.037 INFO:teuthology.orchestra.run.smithi155.stdout:instance: client.14546 (3)172.21.15.155:0/2295869064 2024-03-24T09:47:07.037 INFO:teuthology.orchestra.run.smithi155.stdout:blocklisted: false 2024-03-24T09:47:07.038 INFO:teuthology.orchestra.run.smithi155.stderr:0+1 records in 2024-03-24T09:47:07.038 INFO:teuthology.orchestra.run.smithi155.stderr:0+1 records out 2024-03-24T09:47:07.038 INFO:teuthology.orchestra.run.smithi155.stderr:72 bytes copied, 2.1255e-05 s, 3.4 MB/s 2024-03-24T09:47:07.039 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-24T09:47:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T09:47:07.045 INFO:teuthology.run_tasks:Running task print... 2024-03-24T09:47:07.056 INFO:teuthology.task.print:**** done client 2024-03-24T09:47:07.057 INFO:teuthology.run_tasks:Running task parallel... 2024-03-24T09:47:07.067 INFO:teuthology.task.parallel:starting parallel... 2024-03-24T09:47:07.067 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-03-24T09:47:07.068 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-24T09:47:07.068 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi022.front.sepia.ceph.com 2024-03-24T09:47:07.069 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -e sha1=21b3bd672163092b4e542f772797fcba02d3d4de -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-03-24T09:47:07.069 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-03-24T09:47:07.069 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-03-24T09:47:07.073 INFO:tasks.workunit:Pulling workunits from ref d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b 2024-03-24T09:47:07.074 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-24T09:47:07.074 INFO:tasks.workunit:timeout=3h 2024-03-24T09:47:07.074 INFO:tasks.workunit:cleanup=True 2024-03-24T09:47:07.074 DEBUG:teuthology.orchestra.run.smithi022:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-24T09:47:07.095 INFO:teuthology.orchestra.run.smithi022.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-24T09:47:07.095 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-24T09:47:07.095 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 7dh/125d Inode: 1 Links: 2 2024-03-24T09:47:07.095 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T09:47:07.095 INFO:teuthology.orchestra.run.smithi022.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-24T09:47:07.095 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-03-24T09:47:07.095 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2024-03-24 09:46:34.789396112 +0000 2024-03-24T09:47:07.095 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2024-03-24 09:47:02.266191412 +0000 2024-03-24T09:47:07.095 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: 2024-03-24 09:46:34.789396112 +0000 2024-03-24T09:47:07.096 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-24T09:47:07.096 DEBUG:teuthology.orchestra.run.smithi022:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-24T09:47:07.167 DEBUG:teuthology.orchestra.run.smithi155:> stat -- /home/ubuntu/cephtest/mnt.1 2024-03-24T09:47:07.183 INFO:teuthology.orchestra.run.smithi155.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-03-24T09:47:07.183 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 1 Blocks: 0 IO Block: 65536 directory 2024-03-24T09:47:07.183 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 7eh/126d Inode: 1 Links: 3 2024-03-24T09:47:07.183 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T09:47:07.183 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-24T09:47:07.183 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-03-24T09:47:07.184 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-03-24 09:47:07.163130844 +0000 2024-03-24T09:47:07.184 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-03-24 09:47:07.163130844 +0000 2024-03-24T09:47:07.184 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-03-24 09:46:34.789396112 +0000 2024-03-24T09:47:07.184 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-03-24T09:47:07.184 DEBUG:teuthology.orchestra.run.smithi155:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-03-24T09:47:07.255 DEBUG:teuthology.orchestra.run.smithi022:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/batrick/ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b 2024-03-24T09:47:07.256 DEBUG:teuthology.orchestra.run.smithi155:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://github.com/batrick/ceph /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b 2024-03-24T09:47:07.272 INFO:tasks.workunit.client.0.smithi022.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-24T09:47:07.300 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:47:07.311 INFO:tasks.workunit.client.1.smithi155.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-03-24T09:47:07.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:07 smithi155 ceph-mon[37016]: pgmap v198: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 170 B/s wr, 2 op/s 2024-03-24T09:47:07.746 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:07 smithi022 ceph-mon[29512]: pgmap v198: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 170 B/s wr, 2 op/s 2024-03-24T09:47:08.645 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:08 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:47:08.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:08.888+0000 7fc025448700 1 -- 172.21.15.22:0/1421072414 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc020100b00 msgr2=0x7fc020100f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:08.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:08.888+0000 7fc025448700 1 --2- 172.21.15.22:0/1421072414 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc020100b00 0x7fc020100f20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc01800b0a0 tx=0x7fc01800b3b0 comp rx=0 tx=0).stop 2024-03-24T09:47:08.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:08.888+0000 7fc025448700 1 -- 172.21.15.22:0/1421072414 shutdown_connections 2024-03-24T09:47:08.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:08.888+0000 7fc025448700 1 --2- 172.21.15.22:0/1421072414 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc020101d00 0x7fc020102160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:08.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:08.888+0000 7fc025448700 1 --2- 172.21.15.22:0/1421072414 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc020100b00 0x7fc020100f20 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:08.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:08.888+0000 7fc025448700 1 -- 172.21.15.22:0/1421072414 >> 172.21.15.22:0/1421072414 conn(0x7fc0200fc0a0 msgr2=0x7fc0200fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:47:08.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:08.888+0000 7fc025448700 1 -- 172.21.15.22:0/1421072414 shutdown_connections 2024-03-24T09:47:08.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:08.888+0000 7fc025448700 1 -- 172.21.15.22:0/1421072414 wait complete. 2024-03-24T09:47:08.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:08.889+0000 7fc025448700 1 Processor -- start 2024-03-24T09:47:08.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:08.889+0000 7fc025448700 1 -- start start 2024-03-24T09:47:08.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:08.889+0000 7fc025448700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc020100b00 0x7fc020196650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:47:08.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:08.889+0000 7fc025448700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc020101d00 0x7fc020196b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:47:08.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:08.889+0000 7fc025448700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0201971b0 con 0x7fc020100b00 2024-03-24T09:47:08.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:08.889+0000 7fc025448700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0201972f0 con 0x7fc020101d00 2024-03-24T09:47:08.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:08.889+0000 7fc01effd700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc020100b00 0x7fc020196650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:47:08.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:08.889+0000 7fc01effd700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc020100b00 0x7fc020196650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:38556/0 (socket says 172.21.15.22:38556) 2024-03-24T09:47:08.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:08.889+0000 7fc01effd700 1 -- 172.21.15.22:0/3145885393 learned_addr learned my addr 172.21.15.22:0/3145885393 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:47:08.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:08.889+0000 7fc01e7fc700 1 --2- 172.21.15.22:0/3145885393 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc020101d00 0x7fc020196b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:47:08.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:08.889+0000 7fc01effd700 1 -- 172.21.15.22:0/3145885393 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc020101d00 msgr2=0x7fc020196b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:08.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:08.890+0000 7fc01effd700 1 --2- 172.21.15.22:0/3145885393 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc020101d00 0x7fc020196b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:08.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:08.890+0000 7fc01effd700 1 -- 172.21.15.22:0/3145885393 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc018009d00 con 0x7fc020100b00 2024-03-24T09:47:08.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:08.890+0000 7fc01effd700 1 --2- 172.21.15.22:0/3145885393 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc020100b00 0x7fc020196650 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc018015040 tx=0x7fc018003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:47:08.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:08.890+0000 7fc017fff700 1 -- 172.21.15.22:0/3145885393 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc018021070 con 0x7fc020100b00 2024-03-24T09:47:08.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:08.890+0000 7fc025448700 1 -- 172.21.15.22:0/3145885393 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc02019bca0 con 0x7fc020100b00 2024-03-24T09:47:08.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:08.890+0000 7fc017fff700 1 -- 172.21.15.22:0/3145885393 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc0180075c0 con 0x7fc020100b00 2024-03-24T09:47:08.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:08.890+0000 7fc017fff700 1 -- 172.21.15.22:0/3145885393 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc01800e040 con 0x7fc020100b00 2024-03-24T09:47:08.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:08.890+0000 7fc025448700 1 -- 172.21.15.22:0/3145885393 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc02019c190 con 0x7fc020100b00 2024-03-24T09:47:08.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:08.891+0000 7fc025448700 1 -- 172.21.15.22:0/3145885393 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc020190700 con 0x7fc020100b00 2024-03-24T09:47:08.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:08.891+0000 7fc017fff700 1 -- 172.21.15.22:0/3145885393 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7fc018004250 con 0x7fc020100b00 2024-03-24T09:47:08.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:08.892+0000 7fc017fff700 1 --2- 172.21.15.22:0/3145885393 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fc00c06cce0 0x7fc00c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:47:08.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:08.892+0000 7fc017fff700 1 -- 172.21.15.22:0/3145885393 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fc01808d5c0 con 0x7fc020100b00 2024-03-24T09:47:08.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:08.892+0000 7fc01e7fc700 1 --2- 172.21.15.22:0/3145885393 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fc00c06cce0 0x7fc00c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:47:08.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:08.892+0000 7fc01e7fc700 1 --2- 172.21.15.22:0/3145885393 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fc00c06cce0 0x7fc00c06f1a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc008006fd0 tx=0x7fc008008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:47:08.896 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:08.897+0000 7fc017fff700 1 -- 172.21.15.22:0/3145885393 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc01805b3d0 con 0x7fc020100b00 2024-03-24T09:47:08.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:08 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:47:09.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:09.098+0000 7fc025448700 1 -- 172.21.15.22:0/3145885393 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7fc02002d100 con 0x7fc020100b00 2024-03-24T09:47:09.097 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:09.098+0000 7fc017fff700 1 -- 172.21.15.22:0/3145885393 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7fc01805af60 con 0x7fc020100b00 2024-03-24T09:47:09.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:09.100+0000 7fc025448700 1 -- 172.21.15.22:0/3145885393 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fc00c06cce0 msgr2=0x7fc00c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:09.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:09.100+0000 7fc025448700 1 --2- 172.21.15.22:0/3145885393 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fc00c06cce0 0x7fc00c06f1a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc008006fd0 tx=0x7fc008008040 comp rx=0 tx=0).stop 2024-03-24T09:47:09.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:09.100+0000 7fc025448700 1 -- 172.21.15.22:0/3145885393 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc020100b00 msgr2=0x7fc020196650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:09.099 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:09.100+0000 7fc025448700 1 --2- 172.21.15.22:0/3145885393 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc020100b00 0x7fc020196650 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc018015040 tx=0x7fc018003ce0 comp rx=0 tx=0).stop 2024-03-24T09:47:09.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:09.100+0000 7fc025448700 1 -- 172.21.15.22:0/3145885393 shutdown_connections 2024-03-24T09:47:09.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:09.100+0000 7fc025448700 1 --2- 172.21.15.22:0/3145885393 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fc00c06cce0 0x7fc00c06f1a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:09.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:09.100+0000 7fc025448700 1 --2- 172.21.15.22:0/3145885393 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc020100b00 0x7fc020196650 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:09.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:09.100+0000 7fc025448700 1 --2- 172.21.15.22:0/3145885393 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc020101d00 0x7fc020196b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:09.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:09.100+0000 7fc025448700 1 -- 172.21.15.22:0/3145885393 >> 172.21.15.22:0/3145885393 conn(0x7fc0200fc0a0 msgr2=0x7fc020105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:47:09.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:09.100+0000 7fc025448700 1 -- 172.21.15.22:0/3145885393 shutdown_connections 2024-03-24T09:47:09.100 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:09.100+0000 7fc025448700 1 -- 172.21.15.22:0/3145885393 wait complete. 2024-03-24T09:47:09.700 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:09 smithi022 ceph-mon[29512]: pgmap v199: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 938 B/s rd, 170 B/s wr, 1 op/s 2024-03-24T09:47:09.701 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -e sha1=21b3bd672163092b4e542f772797fcba02d3d4de -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-03-24T09:47:09.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:09 smithi155 ceph-mon[37016]: pgmap v199: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 938 B/s rd, 170 B/s wr, 1 op/s 2024-03-24T09:47:09.972 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:47:11.249 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:11 smithi022 ceph-mon[29512]: pgmap v200: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 938 B/s rd, 170 B/s wr, 1 op/s 2024-03-24T09:47:11.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:11 smithi155 ceph-mon[37016]: pgmap v200: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 938 B/s rd, 170 B/s wr, 1 op/s 2024-03-24T09:47:11.784 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:11.781+0000 7f9e3d8a5700 1 -- 172.21.15.22:0/3049118538 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9e381015d0 msgr2=0x7f9e381039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:11.784 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:11.781+0000 7f9e3d8a5700 1 --2- 172.21.15.22:0/3049118538 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9e381015d0 0x7f9e381039c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f9e28009910 tx=0x7f9e28009c20 comp rx=0 tx=0).stop 2024-03-24T09:47:11.784 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:11.781+0000 7f9e3d8a5700 1 -- 172.21.15.22:0/3049118538 shutdown_connections 2024-03-24T09:47:11.784 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:11.781+0000 7f9e3d8a5700 1 --2- 172.21.15.22:0/3049118538 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e38103f00 0x7f9e381062f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:11.784 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:11.781+0000 7f9e3d8a5700 1 --2- 172.21.15.22:0/3049118538 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9e381015d0 0x7f9e381039c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:11.784 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:11.781+0000 7f9e3d8a5700 1 -- 172.21.15.22:0/3049118538 >> 172.21.15.22:0/3049118538 conn(0x7f9e380faf80 msgr2=0x7f9e380fd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:47:11.784 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:11.781+0000 7f9e3d8a5700 1 -- 172.21.15.22:0/3049118538 shutdown_connections 2024-03-24T09:47:11.785 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:11.781+0000 7f9e3d8a5700 1 -- 172.21.15.22:0/3049118538 wait complete. 2024-03-24T09:47:11.785 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:11.782+0000 7f9e3d8a5700 1 Processor -- start 2024-03-24T09:47:11.785 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:11.782+0000 7f9e3d8a5700 1 -- start start 2024-03-24T09:47:11.785 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:11.783+0000 7f9e3d8a5700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e381015d0 0x7f9e38100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:47:11.785 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:11.783+0000 7f9e3d8a5700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9e38103f00 0x7f9e380ff580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:47:11.785 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:11.783+0000 7f9e3d8a5700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e380ffac0 con 0x7f9e38103f00 2024-03-24T09:47:11.785 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:11.783+0000 7f9e3d8a5700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e380ffc00 con 0x7f9e381015d0 2024-03-24T09:47:11.785 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:11.783+0000 7f9e36ffd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e381015d0 0x7f9e38100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:47:11.785 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:11.783+0000 7f9e367fc700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9e38103f00 0x7f9e380ff580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:47:11.786 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:11.783+0000 7f9e36ffd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e381015d0 0x7f9e38100ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:44096/0 (socket says 172.21.15.22:44096) 2024-03-24T09:47:11.786 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:11.783+0000 7f9e367fc700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9e38103f00 0x7f9e380ff580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:38572/0 (socket says 172.21.15.22:38572) 2024-03-24T09:47:11.786 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:11.783+0000 7f9e36ffd700 1 -- 172.21.15.22:0/10050488 learned_addr learned my addr 172.21.15.22:0/10050488 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:47:11.786 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:11.783+0000 7f9e36ffd700 1 -- 172.21.15.22:0/10050488 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9e38103f00 msgr2=0x7f9e380ff580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:11.786 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:11.783+0000 7f9e36ffd700 1 --2- 172.21.15.22:0/10050488 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9e38103f00 0x7f9e380ff580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:11.786 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:11.783+0000 7f9e36ffd700 1 -- 172.21.15.22:0/10050488 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e280095f0 con 0x7f9e381015d0 2024-03-24T09:47:11.786 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:11.783+0000 7f9e36ffd700 1 --2- 172.21.15.22:0/10050488 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e381015d0 0x7f9e38100ee0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9e2800b800 tx=0x7f9e2800b600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:47:11.786 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:11.784+0000 7f9e3c8a3700 1 -- 172.21.15.22:0/10050488 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e2801d070 con 0x7f9e381015d0 2024-03-24T09:47:11.786 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:11.784+0000 7f9e3c8a3700 1 -- 172.21.15.22:0/10050488 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9e280052a0 con 0x7f9e381015d0 2024-03-24T09:47:11.786 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:11.784+0000 7f9e3d8a5700 1 -- 172.21.15.22:0/10050488 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e380ffe80 con 0x7f9e381015d0 2024-03-24T09:47:11.787 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:11.784+0000 7f9e3c8a3700 1 -- 172.21.15.22:0/10050488 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e28022470 con 0x7f9e381015d0 2024-03-24T09:47:11.787 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:11.784+0000 7f9e3d8a5700 1 -- 172.21.15.22:0/10050488 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e381003f0 con 0x7f9e381015d0 2024-03-24T09:47:11.787 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:11.785+0000 7f9e3d8a5700 1 -- 172.21.15.22:0/10050488 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e3804ea80 con 0x7f9e381015d0 2024-03-24T09:47:11.788 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:11.786+0000 7f9e3c8a3700 1 -- 172.21.15.22:0/10050488 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f9e2800f810 con 0x7f9e381015d0 2024-03-24T09:47:11.788 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:11.786+0000 7f9e3c8a3700 1 --2- 172.21.15.22:0/10050488 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f9e2006cce0 0x7f9e2006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:47:11.788 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:11.786+0000 7f9e3c8a3700 1 -- 172.21.15.22:0/10050488 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f9e2808d4a0 con 0x7f9e381015d0 2024-03-24T09:47:11.788 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:11.786+0000 7f9e367fc700 1 --2- 172.21.15.22:0/10050488 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f9e2006cce0 0x7f9e2006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:47:11.788 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:11.787+0000 7f9e367fc700 1 --2- 172.21.15.22:0/10050488 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f9e2006cce0 0x7f9e2006f1a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c009c60 tx=0x7f9e2c009260 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:47:11.790 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:11.791+0000 7f9e3c8a3700 1 -- 172.21.15.22:0/10050488 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9e2805b320 con 0x7f9e381015d0 2024-03-24T09:47:12.023 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:12.024+0000 7f9e3d8a5700 1 -- 172.21.15.22:0/10050488 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f9e3819e500 con 0x7f9e381015d0 2024-03-24T09:47:12.024 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:12.025+0000 7f9e3c8a3700 1 -- 172.21.15.22:0/10050488 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7f9e2805aeb0 con 0x7f9e381015d0 2024-03-24T09:47:12.026 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:12.027+0000 7f9e3d8a5700 1 -- 172.21.15.22:0/10050488 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f9e2006cce0 msgr2=0x7f9e2006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:12.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:12.027+0000 7f9e3d8a5700 1 --2- 172.21.15.22:0/10050488 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f9e2006cce0 0x7f9e2006f1a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c009c60 tx=0x7f9e2c009260 comp rx=0 tx=0).stop 2024-03-24T09:47:12.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:12.027+0000 7f9e3d8a5700 1 -- 172.21.15.22:0/10050488 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e381015d0 msgr2=0x7f9e38100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:12.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:12.027+0000 7f9e3d8a5700 1 --2- 172.21.15.22:0/10050488 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e381015d0 0x7f9e38100ee0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9e2800b800 tx=0x7f9e2800b600 comp rx=0 tx=0).stop 2024-03-24T09:47:12.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:12.027+0000 7f9e3d8a5700 1 -- 172.21.15.22:0/10050488 shutdown_connections 2024-03-24T09:47:12.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:12.027+0000 7f9e3d8a5700 1 --2- 172.21.15.22:0/10050488 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f9e2006cce0 0x7f9e2006f1a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:12.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:12.027+0000 7f9e3d8a5700 1 --2- 172.21.15.22:0/10050488 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e381015d0 0x7f9e38100ee0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:12.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:12.027+0000 7f9e3d8a5700 1 --2- 172.21.15.22:0/10050488 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9e38103f00 0x7f9e380ff580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:12.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:12.027+0000 7f9e3d8a5700 1 -- 172.21.15.22:0/10050488 >> 172.21.15.22:0/10050488 conn(0x7f9e380faf80 msgr2=0x7f9e380fd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:47:12.027 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:12.027+0000 7f9e3d8a5700 1 -- 172.21.15.22:0/10050488 shutdown_connections 2024-03-24T09:47:12.028 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:12.027+0000 7f9e3d8a5700 1 -- 172.21.15.22:0/10050488 wait complete. 2024-03-24T09:47:12.736 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -e sha1=21b3bd672163092b4e542f772797fcba02d3d4de -- bash -c 'ceph config set global log_to_journald false --force' 2024-03-24T09:47:12.962 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:47:13.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:13 smithi155 ceph-mon[37016]: pgmap v201: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.9 KiB/s rd, 853 B/s wr, 3 op/s 2024-03-24T09:47:14.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:13 smithi022 ceph-mon[29512]: pgmap v201: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.9 KiB/s rd, 853 B/s wr, 3 op/s 2024-03-24T09:47:14.868 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:14.869+0000 7f68ad245700 1 -- 172.21.15.22:0/3800360573 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f68a8103ed0 msgr2=0x7f68a81062c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:14.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:14.869+0000 7f68ad245700 1 --2- 172.21.15.22:0/3800360573 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f68a8103ed0 0x7f68a81062c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f689c009910 tx=0x7f689c009c20 comp rx=0 tx=0).stop 2024-03-24T09:47:14.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:14.869+0000 7f68ad245700 1 -- 172.21.15.22:0/3800360573 shutdown_connections 2024-03-24T09:47:14.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:14.869+0000 7f68ad245700 1 --2- 172.21.15.22:0/3800360573 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f68a8103ed0 0x7f68a81062c0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:14.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:14.869+0000 7f68ad245700 1 --2- 172.21.15.22:0/3800360573 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68a81015a0 0x7f68a8103990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:14.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:14.869+0000 7f68ad245700 1 -- 172.21.15.22:0/3800360573 >> 172.21.15.22:0/3800360573 conn(0x7f68a80fafb0 msgr2=0x7f68a80fd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:47:14.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:14.869+0000 7f68ad245700 1 -- 172.21.15.22:0/3800360573 shutdown_connections 2024-03-24T09:47:14.869 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:14.869+0000 7f68ad245700 1 -- 172.21.15.22:0/3800360573 wait complete. 2024-03-24T09:47:14.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:14.870+0000 7f68ad245700 1 Processor -- start 2024-03-24T09:47:14.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:14.870+0000 7f68ad245700 1 -- start start 2024-03-24T09:47:14.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:14.870+0000 7f68ad245700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f68a81015a0 0x7f68a8100e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:47:14.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:14.870+0000 7f68ad245700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68a8103ed0 0x7f68a80ff550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:47:14.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:14.870+0000 7f68ad245700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a80ffa90 con 0x7f68a81015a0 2024-03-24T09:47:14.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:14.870+0000 7f68ad245700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a80ffbd0 con 0x7f68a8103ed0 2024-03-24T09:47:14.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:14.870+0000 7f68a6d9d700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f68a81015a0 0x7f68a8100e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:47:14.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:14.870+0000 7f68a6d9d700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f68a81015a0 0x7f68a8100e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:38590/0 (socket says 172.21.15.22:38590) 2024-03-24T09:47:14.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:14.870+0000 7f68a6d9d700 1 -- 172.21.15.22:0/2552492917 learned_addr learned my addr 172.21.15.22:0/2552492917 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:47:14.871 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:14.870+0000 7f68a659c700 1 --2- 172.21.15.22:0/2552492917 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68a8103ed0 0x7f68a80ff550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:47:14.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:14.871+0000 7f68a6d9d700 1 -- 172.21.15.22:0/2552492917 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68a8103ed0 msgr2=0x7f68a80ff550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:14.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:14.871+0000 7f68a6d9d700 1 --2- 172.21.15.22:0/2552492917 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68a8103ed0 0x7f68a80ff550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:14.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:14.871+0000 7f68a6d9d700 1 -- 172.21.15.22:0/2552492917 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f689c0095f0 con 0x7f68a81015a0 2024-03-24T09:47:14.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:14.871+0000 7f68a6d9d700 1 --2- 172.21.15.22:0/2552492917 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f68a81015a0 0x7f68a8100e60 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f689800e920 tx=0x7f689800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:47:14.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:14.871+0000 7f688ffff700 1 -- 172.21.15.22:0/2552492917 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f689800c9d0 con 0x7f68a81015a0 2024-03-24T09:47:14.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:14.871+0000 7f68ad245700 1 -- 172.21.15.22:0/2552492917 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68a80ffeb0 con 0x7f68a81015a0 2024-03-24T09:47:14.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:14.871+0000 7f68ad245700 1 -- 172.21.15.22:0/2552492917 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68a819e660 con 0x7f68a81015a0 2024-03-24T09:47:14.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:14.871+0000 7f688ffff700 1 -- 172.21.15.22:0/2552492917 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6898010430 con 0x7f68a81015a0 2024-03-24T09:47:14.872 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:14.872+0000 7f688ffff700 1 -- 172.21.15.22:0/2552492917 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6898018630 con 0x7f68a81015a0 2024-03-24T09:47:14.873 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:14.872+0000 7f68ad245700 1 -- 172.21.15.22:0/2552492917 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68a804ea80 con 0x7f68a81015a0 2024-03-24T09:47:14.873 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:14.873+0000 7f688ffff700 1 -- 172.21.15.22:0/2552492917 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f689800cb30 con 0x7f68a81015a0 2024-03-24T09:47:14.873 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:14.873+0000 7f688ffff700 1 --2- 172.21.15.22:0/2552492917 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f689006cce0 0x7f689006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:47:14.873 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:14.873+0000 7f688ffff700 1 -- 172.21.15.22:0/2552492917 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f6898014070 con 0x7f68a81015a0 2024-03-24T09:47:14.873 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:14.873+0000 7f68a659c700 1 --2- 172.21.15.22:0/2552492917 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f689006cce0 0x7f689006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:47:14.873 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:14.874+0000 7f68a659c700 1 --2- 172.21.15.22:0/2552492917 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f689006cce0 0x7f689006f1a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f689c00b800 tx=0x7f689c009f70 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:47:14.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:14.878+0000 7f688ffff700 1 -- 172.21.15.22:0/2552492917 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f68980529f0 con 0x7f68a81015a0 2024-03-24T09:47:15.083 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:15.084+0000 7f68ad245700 1 -- 172.21.15.22:0/2552492917 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7f68a819e7f0 con 0x7f68a81015a0 2024-03-24T09:47:15.083 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:15.084+0000 7f688ffff700 1 -- 172.21.15.22:0/2552492917 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7f689805a320 con 0x7f68a81015a0 2024-03-24T09:47:15.085 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:15.086+0000 7f68ad245700 1 -- 172.21.15.22:0/2552492917 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f689006cce0 msgr2=0x7f689006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:15.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:15.086+0000 7f68ad245700 1 --2- 172.21.15.22:0/2552492917 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f689006cce0 0x7f689006f1a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f689c00b800 tx=0x7f689c009f70 comp rx=0 tx=0).stop 2024-03-24T09:47:15.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:15.086+0000 7f68ad245700 1 -- 172.21.15.22:0/2552492917 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f68a81015a0 msgr2=0x7f68a8100e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:15.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:15.086+0000 7f68ad245700 1 --2- 172.21.15.22:0/2552492917 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f68a81015a0 0x7f68a8100e60 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f689800e920 tx=0x7f689800ec30 comp rx=0 tx=0).stop 2024-03-24T09:47:15.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:15.087+0000 7f68ad245700 1 -- 172.21.15.22:0/2552492917 shutdown_connections 2024-03-24T09:47:15.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:15.087+0000 7f68ad245700 1 --2- 172.21.15.22:0/2552492917 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f689006cce0 0x7f689006f1a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:15.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:15.087+0000 7f68ad245700 1 --2- 172.21.15.22:0/2552492917 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f68a81015a0 0x7f68a8100e60 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:15.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:15.087+0000 7f68ad245700 1 --2- 172.21.15.22:0/2552492917 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68a8103ed0 0x7f68a80ff550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:15.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:15.087+0000 7f68ad245700 1 -- 172.21.15.22:0/2552492917 >> 172.21.15.22:0/2552492917 conn(0x7f68a80fafb0 msgr2=0x7f68a80fd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:47:15.086 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:15.087+0000 7f68ad245700 1 -- 172.21.15.22:0/2552492917 shutdown_connections 2024-03-24T09:47:15.087 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:15.087+0000 7f68ad245700 1 -- 172.21.15.22:0/2552492917 wait complete. 2024-03-24T09:47:15.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:15 smithi022 ceph-mon[29512]: pgmap v202: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 938 B/s wr, 2 op/s 2024-03-24T09:47:15.908 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:15 smithi155 ceph-mon[37016]: pgmap v202: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.5 KiB/s rd, 938 B/s wr, 2 op/s 2024-03-24T09:47:15.952 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -e sha1=21b3bd672163092b4e542f772797fcba02d3d4de -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-03-24T09:47:16.186 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:47:17.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:17 smithi022 ceph-mon[29512]: pgmap v203: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.9 KiB/s rd, 938 B/s wr, 3 op/s 2024-03-24T09:47:17.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:17 smithi155 ceph-mon[37016]: pgmap v203: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.9 KiB/s rd, 938 B/s wr, 3 op/s 2024-03-24T09:47:18.068 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:18.068+0000 7f3f436fa700 1 -- 172.21.15.22:0/4039691092 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3f3c100ae0 msgr2=0x7f3f3c100f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:18.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:18.068+0000 7f3f436fa700 1 --2- 172.21.15.22:0/4039691092 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3f3c100ae0 0x7f3f3c100f00 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f3f38009910 tx=0x7f3f38009c20 comp rx=0 tx=0).stop 2024-03-24T09:47:18.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:18.068+0000 7f3f436fa700 1 -- 172.21.15.22:0/4039691092 shutdown_connections 2024-03-24T09:47:18.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:18.068+0000 7f3f436fa700 1 --2- 172.21.15.22:0/4039691092 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f3c101ce0 0x7f3f3c102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:18.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:18.068+0000 7f3f436fa700 1 --2- 172.21.15.22:0/4039691092 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3f3c100ae0 0x7f3f3c100f00 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:18.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:18.068+0000 7f3f436fa700 1 -- 172.21.15.22:0/4039691092 >> 172.21.15.22:0/4039691092 conn(0x7f3f3c0fc0a0 msgr2=0x7f3f3c0fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:47:18.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:18.068+0000 7f3f436fa700 1 -- 172.21.15.22:0/4039691092 shutdown_connections 2024-03-24T09:47:18.069 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:18.068+0000 7f3f436fa700 1 -- 172.21.15.22:0/4039691092 wait complete. 2024-03-24T09:47:18.071 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:18.069+0000 7f3f436fa700 1 Processor -- start 2024-03-24T09:47:18.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:18.069+0000 7f3f436fa700 1 -- start start 2024-03-24T09:47:18.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:18.069+0000 7f3f436fa700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f3c100ae0 0x7f3f3c1927e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:47:18.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:18.069+0000 7f3f436fa700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3f3c101ce0 0x7f3f3c192d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:47:18.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:18.069+0000 7f3f436fa700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c18f320 con 0x7f3f3c101ce0 2024-03-24T09:47:18.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:18.069+0000 7f3f436fa700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c18f460 con 0x7f3f3c100ae0 2024-03-24T09:47:18.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:18.070+0000 7f3f40c95700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3f3c101ce0 0x7f3f3c192d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:47:18.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:18.070+0000 7f3f40c95700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3f3c101ce0 0x7f3f3c192d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:38602/0 (socket says 172.21.15.22:38602) 2024-03-24T09:47:18.072 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:18.070+0000 7f3f40c95700 1 -- 172.21.15.22:0/1041717314 learned_addr learned my addr 172.21.15.22:0/1041717314 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:47:18.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:18.070+0000 7f3f41496700 1 --2- 172.21.15.22:0/1041717314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f3c100ae0 0x7f3f3c1927e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:47:18.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:18.070+0000 7f3f40c95700 1 -- 172.21.15.22:0/1041717314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f3c100ae0 msgr2=0x7f3f3c1927e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:18.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:18.070+0000 7f3f40c95700 1 --2- 172.21.15.22:0/1041717314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f3c100ae0 0x7f3f3c1927e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:18.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:18.070+0000 7f3f40c95700 1 -- 172.21.15.22:0/1041717314 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f380095f0 con 0x7f3f3c101ce0 2024-03-24T09:47:18.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:18.070+0000 7f3f40c95700 1 --2- 172.21.15.22:0/1041717314 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3f3c101ce0 0x7f3f3c192d20 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c00d900 tx=0x7f3f2c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:47:18.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:18.070+0000 7f3f327fc700 1 -- 172.21.15.22:0/1041717314 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f2c0041f0 con 0x7f3f3c101ce0 2024-03-24T09:47:18.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:18.070+0000 7f3f327fc700 1 -- 172.21.15.22:0/1041717314 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3f2c004cf0 con 0x7f3f3c101ce0 2024-03-24T09:47:18.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:18.071+0000 7f3f436fa700 1 -- 172.21.15.22:0/1041717314 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f3c18f740 con 0x7f3f3c101ce0 2024-03-24T09:47:18.073 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:18.071+0000 7f3f327fc700 1 -- 172.21.15.22:0/1041717314 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f2c00b750 con 0x7f3f3c101ce0 2024-03-24T09:47:18.074 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:18.071+0000 7f3f436fa700 1 -- 172.21.15.22:0/1041717314 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f3c18fbd0 con 0x7f3f3c101ce0 2024-03-24T09:47:18.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:18.072+0000 7f3f436fa700 1 -- 172.21.15.22:0/1041717314 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f3c1066b0 con 0x7f3f3c101ce0 2024-03-24T09:47:18.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:18.072+0000 7f3f327fc700 1 -- 172.21.15.22:0/1041717314 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f3f2c004750 con 0x7f3f3c101ce0 2024-03-24T09:47:18.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:18.073+0000 7f3f327fc700 1 --2- 172.21.15.22:0/1041717314 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f3f2806cce0 0x7f3f2806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:47:18.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:18.073+0000 7f3f327fc700 1 -- 172.21.15.22:0/1041717314 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f3f2c08c130 con 0x7f3f3c101ce0 2024-03-24T09:47:18.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:18.073+0000 7f3f41496700 1 --2- 172.21.15.22:0/1041717314 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f3f2806cce0 0x7f3f2806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:47:18.075 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:18.074+0000 7f3f41496700 1 --2- 172.21.15.22:0/1041717314 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f3f2806cce0 0x7f3f2806f1a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3f3800b5c0 tx=0x7f3f38005960 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:47:18.078 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:18.077+0000 7f3f327fc700 1 -- 172.21.15.22:0/1041717314 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3f2c059fb0 con 0x7f3f3c101ce0 2024-03-24T09:47:18.340 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:18.340+0000 7f3f436fa700 1 -- 172.21.15.22:0/1041717314 --> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de", "daemon_types": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f3f3c02d120 con 0x7f3f2806cce0 2024-03-24T09:47:19.855 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:19 smithi022 ceph-mon[29512]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:47:19.855 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:19 smithi022 ceph-mon[29512]: pgmap v204: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-24T09:47:19.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:19 smithi155 ceph-mon[37016]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:47:19.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:19 smithi155 ceph-mon[37016]: pgmap v204: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-24T09:47:21.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:21 smithi155 ceph-mon[37016]: pgmap v205: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-24T09:47:21.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:21 smithi022 ceph-mon[29512]: pgmap v205: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-24T09:47:23.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:23 smithi155 ceph-mon[37016]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-24T09:47:23.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:23 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:47:24.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:23 smithi022 ceph-mon[29512]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-24T09:47:24.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:23 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:47:25.846 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:25 smithi022 ceph-mon[29512]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-03-24T09:47:25.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:25 smithi155 ceph-mon[37016]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-03-24T09:47:27.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:27 smithi155 ceph-mon[37016]: pgmap v208: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T09:47:28.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:27 smithi022 ceph-mon[29512]: pgmap v208: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T09:47:29.855 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:29 smithi022 ceph-mon[29512]: pgmap v209: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-03-24T09:47:29.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:29 smithi155 ceph-mon[37016]: pgmap v209: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-03-24T09:47:31.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:31 smithi155 ceph-mon[37016]: pgmap v210: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-03-24T09:47:31.531 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:31 smithi022 ceph-mon[29512]: pgmap v210: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-03-24T09:47:33.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:33 smithi155 ceph-mon[37016]: pgmap v211: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T09:47:34.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:33 smithi022 ceph-mon[29512]: pgmap v211: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T09:47:35.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:35 smithi155 ceph-mon[37016]: pgmap v212: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-03-24T09:47:36.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:35 smithi022 ceph-mon[29512]: pgmap v212: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-03-24T09:47:37.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:37 smithi155 ceph-mon[37016]: pgmap v213: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T09:47:38.031 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:37 smithi022 ceph-mon[29512]: pgmap v213: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T09:47:38.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:38 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:47:39.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:38 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:47:39.855 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:39 smithi022 ceph-mon[29512]: pgmap v214: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-03-24T09:47:39.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:39 smithi155 ceph-mon[37016]: pgmap v214: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-03-24T09:47:41.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:41 smithi155 ceph-mon[37016]: pgmap v215: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-03-24T09:47:41.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:41 smithi022 ceph-mon[29512]: pgmap v215: 65 pgs: 65 active+clean; 584 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-03-24T09:47:43.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:43 smithi155 ceph-mon[37016]: pgmap v216: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T09:47:44.031 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:43 smithi022 ceph-mon[29512]: pgmap v216: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-24T09:47:45.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:45 smithi022 ceph-mon[29512]: pgmap v217: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 341 B/s wr, 2 op/s 2024-03-24T09:47:45.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:45 smithi155 ceph-mon[37016]: pgmap v217: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 341 B/s wr, 2 op/s 2024-03-24T09:47:47.722 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:47 smithi022 ceph-mon[29512]: pgmap v218: 65 pgs: 65 active+clean; 588 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-24T09:47:47.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:47 smithi155 ceph-mon[37016]: pgmap v218: 65 pgs: 65 active+clean; 588 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-24T09:47:48.430 INFO:tasks.workunit.client.1.smithi155.stderr:Note: switching to 'd39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b'. 2024-03-24T09:47:48.430 INFO:tasks.workunit.client.1.smithi155.stderr: 2024-03-24T09:47:48.430 INFO:tasks.workunit.client.1.smithi155.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-24T09:47:48.430 INFO:tasks.workunit.client.1.smithi155.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-24T09:47:48.430 INFO:tasks.workunit.client.1.smithi155.stderr:state without impacting any branches by switching back to a branch. 2024-03-24T09:47:48.430 INFO:tasks.workunit.client.1.smithi155.stderr: 2024-03-24T09:47:48.430 INFO:tasks.workunit.client.1.smithi155.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-24T09:47:48.431 INFO:tasks.workunit.client.1.smithi155.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-24T09:47:48.431 INFO:tasks.workunit.client.1.smithi155.stderr: 2024-03-24T09:47:48.431 INFO:tasks.workunit.client.1.smithi155.stderr: git switch -c 2024-03-24T09:47:48.431 INFO:tasks.workunit.client.1.smithi155.stderr: 2024-03-24T09:47:48.431 INFO:tasks.workunit.client.1.smithi155.stderr:Or undo this operation with: 2024-03-24T09:47:48.431 INFO:tasks.workunit.client.1.smithi155.stderr: 2024-03-24T09:47:48.431 INFO:tasks.workunit.client.1.smithi155.stderr: git switch - 2024-03-24T09:47:48.431 INFO:tasks.workunit.client.1.smithi155.stderr: 2024-03-24T09:47:48.431 INFO:tasks.workunit.client.1.smithi155.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-24T09:47:48.431 INFO:tasks.workunit.client.1.smithi155.stderr: 2024-03-24T09:47:48.432 INFO:tasks.workunit.client.1.smithi155.stderr:HEAD is now at d39ab7bb1f1 qa/suites/fs/workload: enable snap_schedule early 2024-03-24T09:47:48.435 DEBUG:teuthology.orchestra.run.smithi155:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-03-24T09:47:48.495 INFO:tasks.workunit.client.1.smithi155.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-24T09:47:48.497 INFO:tasks.workunit.client.1.smithi155.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-03-24T09:47:48.497 INFO:tasks.workunit.client.1.smithi155.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-24T09:47:48.753 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:48.752+0000 7f3f327fc700 1 -- 172.21.15.22:0/1041717314 <== mgr.14215 v2:172.21.15.22:6800/773863266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f3f3c02d120 con 0x7f3f2806cce0 2024-03-24T09:47:48.753 INFO:teuthology.orchestra.run.smithi022.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T09:47:48.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:48.755+0000 7f3f436fa700 1 -- 172.21.15.22:0/1041717314 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f3f2806cce0 msgr2=0x7f3f2806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:48.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:48.755+0000 7f3f436fa700 1 --2- 172.21.15.22:0/1041717314 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f3f2806cce0 0x7f3f2806f1a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3f3800b5c0 tx=0x7f3f38005960 comp rx=0 tx=0).stop 2024-03-24T09:47:48.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:48.755+0000 7f3f436fa700 1 -- 172.21.15.22:0/1041717314 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3f3c101ce0 msgr2=0x7f3f3c192d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:48.756 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:48.755+0000 7f3f436fa700 1 --2- 172.21.15.22:0/1041717314 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3f3c101ce0 0x7f3f3c192d20 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c00d900 tx=0x7f3f2c00dc10 comp rx=0 tx=0).stop 2024-03-24T09:47:48.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:48.756+0000 7f3f436fa700 1 -- 172.21.15.22:0/1041717314 shutdown_connections 2024-03-24T09:47:48.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:48.756+0000 7f3f436fa700 1 --2- 172.21.15.22:0/1041717314 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f3f2806cce0 0x7f3f2806f1a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:48.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:48.756+0000 7f3f436fa700 1 --2- 172.21.15.22:0/1041717314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f3c100ae0 0x7f3f3c1927e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:48.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:48.756+0000 7f3f436fa700 1 --2- 172.21.15.22:0/1041717314 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3f3c101ce0 0x7f3f3c192d20 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:48.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:48.756+0000 7f3f436fa700 1 -- 172.21.15.22:0/1041717314 >> 172.21.15.22:0/1041717314 conn(0x7f3f3c0fc0a0 msgr2=0x7f3f3c073330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:47:48.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:48.756+0000 7f3f436fa700 1 -- 172.21.15.22:0/1041717314 shutdown_connections 2024-03-24T09:47:48.757 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:48.756+0000 7f3f436fa700 1 -- 172.21.15.22:0/1041717314 wait complete. 2024-03-24T09:47:49.234 INFO:tasks.workunit.client.1.smithi155.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-24T09:47:49.297 INFO:tasks.workunit.client.1.smithi155.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-24T09:47:49.337 INFO:tasks.workunit.client.1.smithi155.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-03-24T09:47:49.339 INFO:tasks.workunit.client.1.smithi155.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-03-24T09:47:49.339 INFO:tasks.workunit.client.1.smithi155.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-24T09:47:49.378 INFO:tasks.workunit.client.1.smithi155.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-03-24T09:47:49.382 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-24T09:47:49.382 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-03-24T09:47:49.455 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-03-24T09:47:49.456 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-03-24T09:47:49.456 DEBUG:teuthology.orchestra.run.smithi155:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-03-24T09:47:49.509 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -e sha1=21b3bd672163092b4e542f772797fcba02d3d4de -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-24T09:47:49.515 INFO:tasks.workunit.client.1.smithi155.stderr:+ mkdir -p fsstress 2024-03-24T09:47:49.516 INFO:tasks.workunit.client.1.smithi155.stderr:+ pushd fsstress 2024-03-24T09:47:49.517 INFO:tasks.workunit.client.1.smithi155.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-03-24T09:47:49.517 INFO:tasks.workunit.client.1.smithi155.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-03-24T09:47:49.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:49 smithi022 ceph-mon[29512]: pgmap v219: 65 pgs: 65 active+clean; 588 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-24T09:47:49.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:49 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:47:49.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:49 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:47:49.817 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:47:49.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:49 smithi155 ceph-mon[37016]: pgmap v219: 65 pgs: 65 active+clean; 588 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-24T09:47:49.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:49 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:47:49.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:49 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:47:50.510 INFO:tasks.workunit.client.1.smithi155.stderr:+ tar xzf ltp-full.tgz 2024-03-24T09:47:50.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:50 smithi022 ceph-mon[29512]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T09:47:50.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:50 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:47:50.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:50 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:47:50.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:50 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:47:50.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:50 smithi155 ceph-mon[37016]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T09:47:50.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:50 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:47:50.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:50 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:47:50.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:50 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:47:51.694 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:51 smithi022 ceph-mon[29512]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T09:47:51.694 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:51 smithi022 ceph-mon[29512]: pgmap v220: 65 pgs: 65 active+clean; 588 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-24T09:47:51.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:51 smithi155 ceph-mon[37016]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T09:47:51.711 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:51 smithi155 ceph-mon[37016]: pgmap v220: 65 pgs: 65 active+clean; 588 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-24T09:47:51.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:51.833+0000 7f631f942700 1 -- 172.21.15.22:0/3908506253 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6318073790 msgr2=0x7f6318073c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:51.834 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:51.833+0000 7f631f942700 1 --2- 172.21.15.22:0/3908506253 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6318073790 0x7f6318073c10 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f6308009910 tx=0x7f6308009c20 comp rx=0 tx=0).stop 2024-03-24T09:47:51.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:51.833+0000 7f631f942700 1 -- 172.21.15.22:0/3908506253 shutdown_connections 2024-03-24T09:47:51.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:51.833+0000 7f631f942700 1 --2- 172.21.15.22:0/3908506253 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6318073790 0x7f6318073c10 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:51.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:51.833+0000 7f631f942700 1 --2- 172.21.15.22:0/3908506253 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6318074d60 0x7f63180731c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:51.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:51.833+0000 7f631f942700 1 -- 172.21.15.22:0/3908506253 >> 172.21.15.22:0/3908506253 conn(0x7f63180fc090 msgr2=0x7f63180fe4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:47:51.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:51.833+0000 7f631f942700 1 -- 172.21.15.22:0/3908506253 shutdown_connections 2024-03-24T09:47:51.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:51.833+0000 7f631f942700 1 -- 172.21.15.22:0/3908506253 wait complete. 2024-03-24T09:47:51.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:51.834+0000 7f631f942700 1 Processor -- start 2024-03-24T09:47:51.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:51.834+0000 7f631f942700 1 -- start start 2024-03-24T09:47:51.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:51.834+0000 7f631f942700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6318073790 0x7f63181986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:47:51.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:51.834+0000 7f631f942700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6318074d60 0x7f6318198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:47:51.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:51.834+0000 7f631f942700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6318199200 con 0x7f6318074d60 2024-03-24T09:47:51.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:51.834+0000 7f631f942700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6318199340 con 0x7f6318073790 2024-03-24T09:47:51.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:51.835+0000 7f631d6de700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6318073790 0x7f63181986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:47:51.835 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:51.835+0000 7f631d6de700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6318073790 0x7f63181986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37252/0 (socket says 172.21.15.22:37252) 2024-03-24T09:47:51.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:51.835+0000 7f631d6de700 1 -- 172.21.15.22:0/2861631679 learned_addr learned my addr 172.21.15.22:0/2861631679 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:47:51.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:51.835+0000 7f631cedd700 1 --2- 172.21.15.22:0/2861631679 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6318074d60 0x7f6318198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:47:51.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:51.835+0000 7f631d6de700 1 -- 172.21.15.22:0/2861631679 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6318074d60 msgr2=0x7f6318198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:51.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:51.835+0000 7f631d6de700 1 --2- 172.21.15.22:0/2861631679 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6318074d60 0x7f6318198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:51.836 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:51.835+0000 7f631d6de700 1 -- 172.21.15.22:0/2861631679 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63080095f0 con 0x7f6318073790 2024-03-24T09:47:51.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:51.835+0000 7f631d6de700 1 --2- 172.21.15.22:0/2861631679 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6318073790 0x7f63181986a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f631400e920 tx=0x7f631400ece0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:47:51.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:51.835+0000 7f630e7fc700 1 -- 172.21.15.22:0/2861631679 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f631400cba0 con 0x7f6318073790 2024-03-24T09:47:51.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:51.835+0000 7f630e7fc700 1 -- 172.21.15.22:0/2861631679 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f631400cd00 con 0x7f6318073790 2024-03-24T09:47:51.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:51.836+0000 7f631f942700 1 -- 172.21.15.22:0/2861631679 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f631819dd30 con 0x7f6318073790 2024-03-24T09:47:51.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:51.836+0000 7f630e7fc700 1 -- 172.21.15.22:0/2861631679 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6314018770 con 0x7f6318073790 2024-03-24T09:47:51.837 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:51.836+0000 7f631f942700 1 -- 172.21.15.22:0/2861631679 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f631819e2a0 con 0x7f6318073790 2024-03-24T09:47:51.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:51.837+0000 7f631f942700 1 -- 172.21.15.22:0/2861631679 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6318192840 con 0x7f6318073790 2024-03-24T09:47:51.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:51.837+0000 7f630e7fc700 1 -- 172.21.15.22:0/2861631679 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f6314010430 con 0x7f6318073790 2024-03-24T09:47:51.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:51.838+0000 7f630e7fc700 1 --2- 172.21.15.22:0/2861631679 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f630406ccf0 0x7f630406f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:47:51.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:51.838+0000 7f630e7fc700 1 -- 172.21.15.22:0/2861631679 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f6314014070 con 0x7f6318073790 2024-03-24T09:47:51.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:51.838+0000 7f631cedd700 1 --2- 172.21.15.22:0/2861631679 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f630406ccf0 0x7f630406f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:47:51.838 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:51.838+0000 7f631cedd700 1 --2- 172.21.15.22:0/2861631679 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f630406ccf0 0x7f630406f1b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f630800b800 tx=0x7f6308009f70 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:47:51.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:51.842+0000 7f630e7fc700 1 -- 172.21.15.22:0/2861631679 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f63140529f0 con 0x7f6318073790 2024-03-24T09:47:52.184 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.183+0000 7f631f942700 1 -- 172.21.15.22:0/2861631679 --> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f63180611f0 con 0x7f630406ccf0 2024-03-24T09:47:52.185 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.185+0000 7f630e7fc700 1 -- 172.21.15.22:0/2861631679 <== mgr.14215 v2:172.21.15.22:6800/773863266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+351 (secure 0 0 0) 0x7f63180611f0 con 0x7f630406ccf0 2024-03-24T09:47:52.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.187+0000 7f631f942700 1 -- 172.21.15.22:0/2861631679 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f630406ccf0 msgr2=0x7f630406f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:52.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.187+0000 7f631f942700 1 --2- 172.21.15.22:0/2861631679 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f630406ccf0 0x7f630406f1b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f630800b800 tx=0x7f6308009f70 comp rx=0 tx=0).stop 2024-03-24T09:47:52.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.187+0000 7f631f942700 1 -- 172.21.15.22:0/2861631679 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6318073790 msgr2=0x7f63181986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:52.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.187+0000 7f631f942700 1 --2- 172.21.15.22:0/2861631679 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6318073790 0x7f63181986a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f631400e920 tx=0x7f631400ece0 comp rx=0 tx=0).stop 2024-03-24T09:47:52.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.188+0000 7f631f942700 1 -- 172.21.15.22:0/2861631679 shutdown_connections 2024-03-24T09:47:52.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.188+0000 7f631f942700 1 --2- 172.21.15.22:0/2861631679 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f630406ccf0 0x7f630406f1b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:52.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.188+0000 7f631f942700 1 --2- 172.21.15.22:0/2861631679 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6318073790 0x7f63181986a0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:52.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.188+0000 7f631f942700 1 --2- 172.21.15.22:0/2861631679 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6318074d60 0x7f6318198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:52.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.188+0000 7f631f942700 1 -- 172.21.15.22:0/2861631679 >> 172.21.15.22:0/2861631679 conn(0x7f63180fc090 msgr2=0x7f6318106ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:47:52.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.188+0000 7f631f942700 1 -- 172.21.15.22:0/2861631679 shutdown_connections 2024-03-24T09:47:52.188 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.188+0000 7f631f942700 1 -- 172.21.15.22:0/2861631679 wait complete. 2024-03-24T09:47:52.202 INFO:teuthology.orchestra.run.smithi022.stdout:true 2024-03-24T09:47:52.354 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.354+0000 7fece1424700 1 -- 172.21.15.22:0/142517520 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecdc10a300 msgr2=0x7fecdc10c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:52.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.354+0000 7fece1424700 1 --2- 172.21.15.22:0/142517520 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecdc10a300 0x7fecdc10c790 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fecd4009f90 tx=0x7fecd400b330 comp rx=0 tx=0).stop 2024-03-24T09:47:52.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.354+0000 7fece1424700 1 -- 172.21.15.22:0/142517520 shutdown_connections 2024-03-24T09:47:52.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.354+0000 7fece1424700 1 --2- 172.21.15.22:0/142517520 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecdc10a300 0x7fecdc10c790 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:52.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.354+0000 7fece1424700 1 --2- 172.21.15.22:0/142517520 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fecdc072ac0 0x7fecdc109dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:52.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.354+0000 7fece1424700 1 -- 172.21.15.22:0/142517520 >> 172.21.15.22:0/142517520 conn(0x7fecdc06da40 msgr2=0x7fecdc06fe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:47:52.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.354+0000 7fece1424700 1 -- 172.21.15.22:0/142517520 shutdown_connections 2024-03-24T09:47:52.355 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.354+0000 7fece1424700 1 -- 172.21.15.22:0/142517520 wait complete. 2024-03-24T09:47:52.356 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.355+0000 7fece1424700 1 Processor -- start 2024-03-24T09:47:52.356 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.355+0000 7fece1424700 1 -- start start 2024-03-24T09:47:52.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.355+0000 7fece1424700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecdc072ac0 0x7fecdc1a52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:47:52.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.355+0000 7fece1424700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fecdc10a300 0x7fecdc1a57f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:47:52.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.355+0000 7fece1424700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecdc1a5e30 con 0x7fecdc10a300 2024-03-24T09:47:52.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.355+0000 7fece1424700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecdc1a5fa0 con 0x7fecdc072ac0 2024-03-24T09:47:52.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.356+0000 7fecda7fc700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fecdc10a300 0x7fecdc1a57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:47:52.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.356+0000 7fecda7fc700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fecdc10a300 0x7fecdc1a57f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42566/0 (socket says 172.21.15.22:42566) 2024-03-24T09:47:52.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.356+0000 7fecda7fc700 1 -- 172.21.15.22:0/4038270712 learned_addr learned my addr 172.21.15.22:0/4038270712 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:47:52.357 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.356+0000 7fecdaffd700 1 --2- 172.21.15.22:0/4038270712 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecdc072ac0 0x7fecdc1a52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:47:52.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.357+0000 7fecda7fc700 1 -- 172.21.15.22:0/4038270712 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecdc072ac0 msgr2=0x7fecdc1a52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:52.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.357+0000 7fecda7fc700 1 --2- 172.21.15.22:0/4038270712 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecdc072ac0 0x7fecdc1a52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:52.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.357+0000 7fecda7fc700 1 -- 172.21.15.22:0/4038270712 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd4009c40 con 0x7fecdc10a300 2024-03-24T09:47:52.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.357+0000 7fecda7fc700 1 --2- 172.21.15.22:0/4038270712 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fecdc10a300 0x7fecdc1a57f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fecd4006070 tx=0x7fecd400bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:47:52.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.358+0000 7fecd3fff700 1 -- 172.21.15.22:0/4038270712 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fecd400e070 con 0x7fecdc10a300 2024-03-24T09:47:52.358 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.358+0000 7fece1424700 1 -- 172.21.15.22:0/4038270712 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecdc077140 con 0x7fecdc10a300 2024-03-24T09:47:52.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.358+0000 7fece1424700 1 -- 172.21.15.22:0/4038270712 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecdc077710 con 0x7fecdc10a300 2024-03-24T09:47:52.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.358+0000 7fecd3fff700 1 -- 172.21.15.22:0/4038270712 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fecd4003de0 con 0x7fecdc10a300 2024-03-24T09:47:52.359 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.358+0000 7fecd3fff700 1 -- 172.21.15.22:0/4038270712 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fecd4012660 con 0x7fecdc10a300 2024-03-24T09:47:52.360 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.360+0000 7fecd3fff700 1 -- 172.21.15.22:0/4038270712 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7fecd4019040 con 0x7fecdc10a300 2024-03-24T09:47:52.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.361+0000 7fece1424700 1 -- 172.21.15.22:0/4038270712 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecbc005320 con 0x7fecdc10a300 2024-03-24T09:47:52.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.361+0000 7fecd3fff700 1 --2- 172.21.15.22:0/4038270712 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fecc806cd40 0x7fecc806f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:47:52.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.361+0000 7fecd3fff700 1 -- 172.21.15.22:0/4038270712 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fecd408d170 con 0x7fecdc10a300 2024-03-24T09:47:52.361 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.361+0000 7fecdaffd700 1 --2- 172.21.15.22:0/4038270712 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fecc806cd40 0x7fecc806f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:47:52.362 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.362+0000 7fecdaffd700 1 --2- 172.21.15.22:0/4038270712 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fecc806cd40 0x7fecc806f200 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fecdc1a67c0 tx=0x7feccc009310 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:47:52.371 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.371+0000 7fecd3fff700 1 -- 172.21.15.22:0/4038270712 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fecd4058450 con 0x7fecdc10a300 2024-03-24T09:47:52.760 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.760+0000 7fece1424700 1 -- 172.21.15.22:0/4038270712 --> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fecbc000bf0 con 0x7fecc806cd40 2024-03-24T09:47:52.761 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.761+0000 7fecd3fff700 1 -- 172.21.15.22:0/4038270712 <== mgr.14215 v2:172.21.15.22:6800/773863266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+351 (secure 0 0 0) 0x7fecbc000bf0 con 0x7fecc806cd40 2024-03-24T09:47:52.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.763+0000 7fece1424700 1 -- 172.21.15.22:0/4038270712 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fecc806cd40 msgr2=0x7fecc806f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:52.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.763+0000 7fece1424700 1 --2- 172.21.15.22:0/4038270712 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fecc806cd40 0x7fecc806f200 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fecdc1a67c0 tx=0x7feccc009310 comp rx=0 tx=0).stop 2024-03-24T09:47:52.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.763+0000 7fece1424700 1 -- 172.21.15.22:0/4038270712 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fecdc10a300 msgr2=0x7fecdc1a57f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:52.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.763+0000 7fece1424700 1 --2- 172.21.15.22:0/4038270712 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fecdc10a300 0x7fecdc1a57f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fecd4006070 tx=0x7fecd400bdc0 comp rx=0 tx=0).stop 2024-03-24T09:47:52.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.763+0000 7fece1424700 1 -- 172.21.15.22:0/4038270712 shutdown_connections 2024-03-24T09:47:52.764 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.763+0000 7fece1424700 1 --2- 172.21.15.22:0/4038270712 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fecc806cd40 0x7fecc806f200 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:52.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.763+0000 7fece1424700 1 --2- 172.21.15.22:0/4038270712 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecdc072ac0 0x7fecdc1a52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:52.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.763+0000 7fece1424700 1 --2- 172.21.15.22:0/4038270712 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fecdc10a300 0x7fecdc1a57f0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:52.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.763+0000 7fece1424700 1 -- 172.21.15.22:0/4038270712 >> 172.21.15.22:0/4038270712 conn(0x7fecdc06da40 msgr2=0x7fecdc10b870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:47:52.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.764+0000 7fece1424700 1 -- 172.21.15.22:0/4038270712 shutdown_connections 2024-03-24T09:47:52.765 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.764+0000 7fece1424700 1 -- 172.21.15.22:0/4038270712 wait complete. 2024-03-24T09:47:52.879 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.879+0000 7faed7461700 1 -- 172.21.15.22:0/3599249417 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faed00ff160 msgr2=0x7faed00ff580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:52.880 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.879+0000 7faed7461700 1 --2- 172.21.15.22:0/3599249417 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faed00ff160 0x7faed00ff580 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7faecc009910 tx=0x7faecc009c20 comp rx=0 tx=0).stop 2024-03-24T09:47:52.880 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.879+0000 7faed7461700 1 -- 172.21.15.22:0/3599249417 shutdown_connections 2024-03-24T09:47:52.880 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.879+0000 7faed7461700 1 --2- 172.21.15.22:0/3599249417 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faed0100360 0x7faed01007c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:52.880 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.879+0000 7faed7461700 1 --2- 172.21.15.22:0/3599249417 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faed00ff160 0x7faed00ff580 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:52.880 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.879+0000 7faed7461700 1 -- 172.21.15.22:0/3599249417 >> 172.21.15.22:0/3599249417 conn(0x7faed00fa700 msgr2=0x7faed00fcb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:47:52.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.880+0000 7faed7461700 1 -- 172.21.15.22:0/3599249417 shutdown_connections 2024-03-24T09:47:52.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.880+0000 7faed7461700 1 -- 172.21.15.22:0/3599249417 wait complete. 2024-03-24T09:47:52.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.880+0000 7faed7461700 1 Processor -- start 2024-03-24T09:47:52.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.880+0000 7faed7461700 1 -- start start 2024-03-24T09:47:52.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.881+0000 7faed7461700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faed0100360 0x7faed01989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:47:52.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.881+0000 7faed7461700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faed0198f10 0x7faed019df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:47:52.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.881+0000 7faed7461700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed0199420 con 0x7faed0100360 2024-03-24T09:47:52.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.881+0000 7faed7461700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed0199590 con 0x7faed0198f10 2024-03-24T09:47:52.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.881+0000 7faed49fc700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faed0198f10 0x7faed019df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:47:52.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.881+0000 7faed49fc700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faed0198f10 0x7faed019df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37284/0 (socket says 172.21.15.22:37284) 2024-03-24T09:47:52.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.881+0000 7faed49fc700 1 -- 172.21.15.22:0/173757119 learned_addr learned my addr 172.21.15.22:0/173757119 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:47:52.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.881+0000 7faed51fd700 1 --2- 172.21.15.22:0/173757119 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faed0100360 0x7faed01989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:47:52.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.881+0000 7faed49fc700 1 -- 172.21.15.22:0/173757119 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faed0100360 msgr2=0x7faed01989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:52.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.881+0000 7faed49fc700 1 --2- 172.21.15.22:0/173757119 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faed0100360 0x7faed01989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:52.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.881+0000 7faed49fc700 1 -- 172.21.15.22:0/173757119 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faecc0095f0 con 0x7faed0198f10 2024-03-24T09:47:52.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.882+0000 7faed49fc700 1 --2- 172.21.15.22:0/173757119 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faed0198f10 0x7faed019df30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7faec000e920 tx=0x7faec000ece0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:47:52.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.882+0000 7faed51fd700 1 --2- 172.21.15.22:0/173757119 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faed0100360 0x7faed01989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T09:47:52.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.882+0000 7faec67fc700 1 -- 172.21.15.22:0/173757119 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faec000cba0 con 0x7faed0198f10 2024-03-24T09:47:52.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.882+0000 7faed7461700 1 -- 172.21.15.22:0/173757119 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faed019e4d0 con 0x7faed0198f10 2024-03-24T09:47:52.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.882+0000 7faed7461700 1 -- 172.21.15.22:0/173757119 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faed019eaa0 con 0x7faed0198f10 2024-03-24T09:47:52.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.882+0000 7faec67fc700 1 -- 172.21.15.22:0/173757119 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faec000cd00 con 0x7faed0198f10 2024-03-24T09:47:52.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.882+0000 7faec67fc700 1 -- 172.21.15.22:0/173757119 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faec0018770 con 0x7faed0198f10 2024-03-24T09:47:52.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.883+0000 7faed7461700 1 -- 172.21.15.22:0/173757119 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faed0104d30 con 0x7faed0198f10 2024-03-24T09:47:52.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.884+0000 7faec67fc700 1 -- 172.21.15.22:0/173757119 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7faec0010430 con 0x7faed0198f10 2024-03-24T09:47:52.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.884+0000 7faec67fc700 1 --2- 172.21.15.22:0/173757119 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7faebc06ccf0 0x7faebc06f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:47:52.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.884+0000 7faec67fc700 1 -- 172.21.15.22:0/173757119 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7faec0014070 con 0x7faed0198f10 2024-03-24T09:47:52.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.884+0000 7faed51fd700 1 --2- 172.21.15.22:0/173757119 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7faebc06ccf0 0x7faebc06f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:47:52.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.885+0000 7faed51fd700 1 --2- 172.21.15.22:0/173757119 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7faebc06ccf0 0x7faebc06f1b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7faecc00b800 tx=0x7faecc00b640 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:47:52.894 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:52.893+0000 7faec67fc700 1 -- 172.21.15.22:0/173757119 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7faec00529f0 con 0x7faed0198f10 2024-03-24T09:47:53.116 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.116+0000 7faed7461700 1 -- 172.21.15.22:0/173757119 --> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7faed00611f0 con 0x7faebc06ccf0 2024-03-24T09:47:53.135 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.134+0000 7faec67fc700 1 -- 172.21.15.22:0/173757119 <== mgr.14215 v2:172.21.15.22:6800/773863266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3792 (secure 0 0 0) 0x7faed00611f0 con 0x7faebc06ccf0 2024-03-24T09:47:53.135 INFO:teuthology.orchestra.run.smithi022.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T09:47:53.135 INFO:teuthology.orchestra.run.smithi022.stdout:alertmanager.smithi022 smithi022 *:9093,9094 running (5m) 53s ago 6m 25.1M - 0.25.0 c8568f914cd2 961b9fd08390 2024-03-24T09:47:53.135 INFO:teuthology.orchestra.run.smithi022.stdout:ceph-exporter.smithi022 smithi022 running (7m) 53s ago 7m 7457k - 18.2.2-952-g738c2fd6 68aff9aff3b2 513e822989e8 2024-03-24T09:47:53.135 INFO:teuthology.orchestra.run.smithi022.stdout:ceph-exporter.smithi155 smithi155 running (6m) 58s ago 6m 7390k - 18.2.2-952-g738c2fd6 68aff9aff3b2 2bc65c464d99 2024-03-24T09:47:53.135 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi022 smithi022 running (7m) 53s ago 7m 6665k - 18.2.2-952-g738c2fd6 68aff9aff3b2 a2a36122163c 2024-03-24T09:47:53.136 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi155 smithi155 running (6m) 58s ago 5m 6665k - 18.2.2-952-g738c2fd6 68aff9aff3b2 2d0b688c3a61 2024-03-24T09:47:53.136 INFO:teuthology.orchestra.run.smithi022.stdout:grafana.smithi022 smithi022 *:3000 running (5m) 53s ago 6m 85.9M - 9.4.7 954c08fa6188 082a79d8dba0 2024-03-24T09:47:53.136 INFO:teuthology.orchestra.run.smithi022.stdout:mds.cephfs.smithi022.mhcbrm smithi022 running (79s) 52s ago 79s 11.3M - 18.2.2-952-g738c2fd6 68aff9aff3b2 e64e89f0270a 2024-03-24T09:47:53.136 INFO:teuthology.orchestra.run.smithi022.stdout:mds.cephfs.smithi022.qbmomh smithi022 running (72s) 52s ago 72s 17.0M - 18.2.2-952-g738c2fd6 68aff9aff3b2 77e763d4929f 2024-03-24T09:47:53.136 INFO:teuthology.orchestra.run.smithi022.stdout:mds.cephfs.smithi155.dakojr smithi155 running (70s) 58s ago 69s 13.1M - 18.2.2-952-g738c2fd6 68aff9aff3b2 279a6ccdd766 2024-03-24T09:47:53.136 INFO:teuthology.orchestra.run.smithi022.stdout:mds.cephfs.smithi155.pwhqcw smithi155 running (77s) 58s ago 76s 16.0M - 18.2.2-952-g738c2fd6 68aff9aff3b2 a3b6d24090c5 2024-03-24T09:47:53.136 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi022.emgmxz smithi022 *:9283,8765,8443 running (8m) 53s ago 8m 496M - 18.2.2-952-g738c2fd6 68aff9aff3b2 9c46d65fb5c7 2024-03-24T09:47:53.136 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi155.itwzew smithi155 *:8443,9283,8765 running (5m) 58s ago 5m 452M - 18.2.2-952-g738c2fd6 68aff9aff3b2 076d66e31d28 2024-03-24T09:47:53.136 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi022 smithi022 running (8m) 53s ago 8m 45.8M 2048M 18.2.2-952-g738c2fd6 68aff9aff3b2 59736938121b 2024-03-24T09:47:53.137 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi155 smithi155 running (5m) 58s ago 5m 35.8M 2048M 18.2.2-952-g738c2fd6 68aff9aff3b2 9188cfee0421 2024-03-24T09:47:53.137 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi022 smithi022 *:9100 running (6m) 53s ago 6m 15.5M - 1.5.0 0da6a335fe13 0734bfdc392d 2024-03-24T09:47:53.137 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi155 smithi155 *:9100 running (5m) 58s ago 5m 14.5M - 1.5.0 0da6a335fe13 18110dca979b 2024-03-24T09:47:53.137 INFO:teuthology.orchestra.run.smithi022.stdout:osd.0 smithi022 running (5m) 52s ago 5m 46.9M 3948M 18.2.2-952-g738c2fd6 68aff9aff3b2 db726da62a1c 2024-03-24T09:47:53.137 INFO:teuthology.orchestra.run.smithi022.stdout:osd.1 smithi022 running (4m) 52s ago 4m 48.5M 3948M 18.2.2-952-g738c2fd6 68aff9aff3b2 3c6af2f5b6f1 2024-03-24T09:47:53.137 INFO:teuthology.orchestra.run.smithi022.stdout:osd.2 smithi022 running (4m) 52s ago 4m 46.6M 3948M 18.2.2-952-g738c2fd6 68aff9aff3b2 99b3c8840e7d 2024-03-24T09:47:53.137 INFO:teuthology.orchestra.run.smithi022.stdout:osd.3 smithi155 running (3m) 58s ago 3m 45.6M 4970M 18.2.2-952-g738c2fd6 68aff9aff3b2 534b4e3f3f15 2024-03-24T09:47:53.137 INFO:teuthology.orchestra.run.smithi022.stdout:osd.4 smithi155 running (3m) 58s ago 3m 43.9M 4970M 18.2.2-952-g738c2fd6 68aff9aff3b2 a269f005acda 2024-03-24T09:47:53.137 INFO:teuthology.orchestra.run.smithi022.stdout:osd.5 smithi155 running (2m) 58s ago 2m 42.8M 4970M 18.2.2-952-g738c2fd6 68aff9aff3b2 20dd2513a0df 2024-03-24T09:47:53.138 INFO:teuthology.orchestra.run.smithi022.stdout:prometheus.smithi022 smithi022 *:9095 running (5m) 52s ago 6m 55.1M - 2.43.0 a07b618ecd1d 5cb914be4e25 2024-03-24T09:47:53.139 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.137+0000 7faed7461700 1 -- 172.21.15.22:0/173757119 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7faebc06ccf0 msgr2=0x7faebc06f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:53.139 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.137+0000 7faed7461700 1 --2- 172.21.15.22:0/173757119 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7faebc06ccf0 0x7faebc06f1b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7faecc00b800 tx=0x7faecc00b640 comp rx=0 tx=0).stop 2024-03-24T09:47:53.139 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.137+0000 7faed7461700 1 -- 172.21.15.22:0/173757119 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faed0198f10 msgr2=0x7faed019df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:53.139 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.137+0000 7faed7461700 1 --2- 172.21.15.22:0/173757119 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faed0198f10 0x7faed019df30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7faec000e920 tx=0x7faec000ece0 comp rx=0 tx=0).stop 2024-03-24T09:47:53.139 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.137+0000 7faed7461700 1 -- 172.21.15.22:0/173757119 shutdown_connections 2024-03-24T09:47:53.139 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.137+0000 7faed7461700 1 --2- 172.21.15.22:0/173757119 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7faebc06ccf0 0x7faebc06f1b0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:53.139 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.137+0000 7faed7461700 1 --2- 172.21.15.22:0/173757119 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faed0100360 0x7faed01989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:53.139 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.137+0000 7faed7461700 1 --2- 172.21.15.22:0/173757119 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faed0198f10 0x7faed019df30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:53.140 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.137+0000 7faed7461700 1 -- 172.21.15.22:0/173757119 >> 172.21.15.22:0/173757119 conn(0x7faed00fa700 msgr2=0x7faed0106b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:47:53.140 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.137+0000 7faed7461700 1 -- 172.21.15.22:0/173757119 shutdown_connections 2024-03-24T09:47:53.140 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.137+0000 7faed7461700 1 -- 172.21.15.22:0/173757119 wait complete. 2024-03-24T09:47:53.248 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.247+0000 7f209bd70700 1 -- 172.21.15.22:0/2873102191 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2094100390 msgr2=0x7f20941007f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:53.248 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.247+0000 7f209bd70700 1 --2- 172.21.15.22:0/2873102191 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2094100390 0x7f20941007f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f2084009910 tx=0x7f2084009c20 comp rx=0 tx=0).stop 2024-03-24T09:47:53.248 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.247+0000 7f209bd70700 1 -- 172.21.15.22:0/2873102191 shutdown_connections 2024-03-24T09:47:53.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.247+0000 7f209bd70700 1 --2- 172.21.15.22:0/2873102191 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2094100390 0x7f20941007f0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:53.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.247+0000 7f209bd70700 1 --2- 172.21.15.22:0/2873102191 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20940ff190 0x7f20940ff5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:53.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.247+0000 7f209bd70700 1 -- 172.21.15.22:0/2873102191 >> 172.21.15.22:0/2873102191 conn(0x7f20940fa730 msgr2=0x7f20940fcb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:47:53.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.248+0000 7f209bd70700 1 -- 172.21.15.22:0/2873102191 shutdown_connections 2024-03-24T09:47:53.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.248+0000 7f209bd70700 1 -- 172.21.15.22:0/2873102191 wait complete. 2024-03-24T09:47:53.249 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.248+0000 7f209bd70700 1 Processor -- start 2024-03-24T09:47:53.251 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.248+0000 7f209bd70700 1 -- start start 2024-03-24T09:47:53.251 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.249+0000 7f209bd70700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f20940ff190 0x7f20941959f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:47:53.251 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.249+0000 7f209bd70700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2094100390 0x7f2094194090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:47:53.251 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.249+0000 7f209bd70700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f209419a120 con 0x7f20940ff190 2024-03-24T09:47:53.251 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.249+0000 7f209bd70700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f209419a260 con 0x7f2094100390 2024-03-24T09:47:53.252 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.249+0000 7f2099b0c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f20940ff190 0x7f20941959f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:47:53.252 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.249+0000 7f209930b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2094100390 0x7f2094194090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:47:53.252 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.249+0000 7f209930b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2094100390 0x7f2094194090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37300/0 (socket says 172.21.15.22:37300) 2024-03-24T09:47:53.252 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.249+0000 7f209930b700 1 -- 172.21.15.22:0/324992418 learned_addr learned my addr 172.21.15.22:0/324992418 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:47:53.252 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.249+0000 7f209930b700 1 -- 172.21.15.22:0/324992418 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f20940ff190 msgr2=0x7f20941959f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:53.252 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.249+0000 7f209930b700 1 --2- 172.21.15.22:0/324992418 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f20940ff190 0x7f20941959f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:53.252 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.249+0000 7f209930b700 1 -- 172.21.15.22:0/324992418 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20840095f0 con 0x7f2094100390 2024-03-24T09:47:53.252 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.249+0000 7f2099b0c700 1 --2- 172.21.15.22:0/324992418 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f20940ff190 0x7f20941959f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:47:53.253 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.250+0000 7f209930b700 1 --2- 172.21.15.22:0/324992418 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2094100390 0x7f2094194090 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f208400b800 tx=0x7f2084005f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:47:53.253 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.250+0000 7f208affd700 1 -- 172.21.15.22:0/324992418 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f208401d070 con 0x7f2094100390 2024-03-24T09:47:53.253 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.250+0000 7f209bd70700 1 -- 172.21.15.22:0/324992418 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20941945d0 con 0x7f2094100390 2024-03-24T09:47:53.253 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.250+0000 7f208affd700 1 -- 172.21.15.22:0/324992418 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f20840052a0 con 0x7f2094100390 2024-03-24T09:47:53.253 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.250+0000 7f208affd700 1 -- 172.21.15.22:0/324992418 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2084017400 con 0x7f2094100390 2024-03-24T09:47:53.253 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.250+0000 7f209bd70700 1 -- 172.21.15.22:0/324992418 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2094194b10 con 0x7f2094100390 2024-03-24T09:47:53.254 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.251+0000 7f209bd70700 1 -- 172.21.15.22:0/324992418 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2094107e30 con 0x7f2094100390 2024-03-24T09:47:53.254 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.252+0000 7f208affd700 1 -- 172.21.15.22:0/324992418 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f2084003780 con 0x7f2094100390 2024-03-24T09:47:53.254 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.252+0000 7f208affd700 1 --2- 172.21.15.22:0/324992418 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f208006cce0 0x7f208006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:47:53.255 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.253+0000 7f208affd700 1 -- 172.21.15.22:0/324992418 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f208408d6a0 con 0x7f2094100390 2024-03-24T09:47:53.255 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.253+0000 7f2099b0c700 1 --2- 172.21.15.22:0/324992418 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f208006cce0 0x7f208006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:47:53.255 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.253+0000 7f2099b0c700 1 --2- 172.21.15.22:0/324992418 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f208006cce0 0x7f208006f1a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f20941001f0 tx=0x7f2090010040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:47:53.262 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.261+0000 7f208affd700 1 -- 172.21.15.22:0/324992418 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2094107e30 con 0x7f2094100390 2024-03-24T09:47:53.657 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.657+0000 7f209bd70700 1 -- 172.21.15.22:0/324992418 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2094107dc0 con 0x7f2094100390 2024-03-24T09:47:53.658 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.658+0000 7f208affd700 1 -- 172.21.15.22:0/324992418 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f208405b480 con 0x7f2094100390 2024-03-24T09:47:53.659 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T09:47:53.659 INFO:teuthology.orchestra.run.smithi022.stdout: "mon": { 2024-03-24T09:47:53.659 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 18.2.2-952-g738c2fd6 (738c2fd67dbc0b17e2e99568a03974712d998229) reef (stable)": 2 2024-03-24T09:47:53.659 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T09:47:53.659 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr": { 2024-03-24T09:47:53.659 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 18.2.2-952-g738c2fd6 (738c2fd67dbc0b17e2e99568a03974712d998229) reef (stable)": 2 2024-03-24T09:47:53.659 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T09:47:53.660 INFO:teuthology.orchestra.run.smithi022.stdout: "osd": { 2024-03-24T09:47:53.660 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 18.2.2-952-g738c2fd6 (738c2fd67dbc0b17e2e99568a03974712d998229) reef (stable)": 6 2024-03-24T09:47:53.660 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T09:47:53.660 INFO:teuthology.orchestra.run.smithi022.stdout: "mds": { 2024-03-24T09:47:53.660 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 18.2.2-952-g738c2fd6 (738c2fd67dbc0b17e2e99568a03974712d998229) reef (stable)": 4 2024-03-24T09:47:53.660 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T09:47:53.660 INFO:teuthology.orchestra.run.smithi022.stdout: "overall": { 2024-03-24T09:47:53.660 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 18.2.2-952-g738c2fd6 (738c2fd67dbc0b17e2e99568a03974712d998229) reef (stable)": 14 2024-03-24T09:47:53.660 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T09:47:53.661 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T09:47:53.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.660+0000 7f209bd70700 1 -- 172.21.15.22:0/324992418 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f208006cce0 msgr2=0x7f208006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:53.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.660+0000 7f209bd70700 1 --2- 172.21.15.22:0/324992418 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f208006cce0 0x7f208006f1a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f20941001f0 tx=0x7f2090010040 comp rx=0 tx=0).stop 2024-03-24T09:47:53.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.660+0000 7f209bd70700 1 -- 172.21.15.22:0/324992418 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2094100390 msgr2=0x7f2094194090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:53.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.660+0000 7f209bd70700 1 --2- 172.21.15.22:0/324992418 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2094100390 0x7f2094194090 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f208400b800 tx=0x7f2084005f00 comp rx=0 tx=0).stop 2024-03-24T09:47:53.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.660+0000 7f209bd70700 1 -- 172.21.15.22:0/324992418 shutdown_connections 2024-03-24T09:47:53.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.660+0000 7f209bd70700 1 --2- 172.21.15.22:0/324992418 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f208006cce0 0x7f208006f1a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:53.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.660+0000 7f209bd70700 1 --2- 172.21.15.22:0/324992418 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f20940ff190 0x7f20941959f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:53.662 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.660+0000 7f209bd70700 1 --2- 172.21.15.22:0/324992418 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2094100390 0x7f2094194090 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:53.663 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.660+0000 7f209bd70700 1 -- 172.21.15.22:0/324992418 >> 172.21.15.22:0/324992418 conn(0x7f20940fa730 msgr2=0x7f20941039b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:47:53.663 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.661+0000 7f209bd70700 1 -- 172.21.15.22:0/324992418 shutdown_connections 2024-03-24T09:47:53.663 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.661+0000 7f209bd70700 1 -- 172.21.15.22:0/324992418 wait complete. 2024-03-24T09:47:53.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.776+0000 7f9c260e1700 1 -- 172.21.15.22:0/344066155 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9c20103f30 msgr2=0x7f9c20106320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:53.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.776+0000 7f9c260e1700 1 --2- 172.21.15.22:0/344066155 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9c20103f30 0x7f9c20106320 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f9c14009910 tx=0x7f9c14009c20 comp rx=0 tx=0).stop 2024-03-24T09:47:53.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.776+0000 7f9c260e1700 1 -- 172.21.15.22:0/344066155 shutdown_connections 2024-03-24T09:47:53.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.776+0000 7f9c260e1700 1 --2- 172.21.15.22:0/344066155 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9c20103f30 0x7f9c20106320 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:53.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.776+0000 7f9c260e1700 1 --2- 172.21.15.22:0/344066155 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c20101600 0x7f9c201039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:53.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.776+0000 7f9c260e1700 1 -- 172.21.15.22:0/344066155 >> 172.21.15.22:0/344066155 conn(0x7f9c200faf70 msgr2=0x7f9c200fd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:47:53.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.776+0000 7f9c260e1700 1 -- 172.21.15.22:0/344066155 shutdown_connections 2024-03-24T09:47:53.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.776+0000 7f9c260e1700 1 -- 172.21.15.22:0/344066155 wait complete. 2024-03-24T09:47:53.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.777+0000 7f9c260e1700 1 Processor -- start 2024-03-24T09:47:53.778 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.777+0000 7f9c260e1700 1 -- start start 2024-03-24T09:47:53.779 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.777+0000 7f9c260e1700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9c20101600 0x7f9c201964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:47:53.779 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.777+0000 7f9c260e1700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c20103f30 0x7f9c201969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:47:53.779 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.777+0000 7f9c260e1700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c20197010 con 0x7f9c20101600 2024-03-24T09:47:53.779 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.777+0000 7f9c260e1700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c20197150 con 0x7f9c20103f30 2024-03-24T09:47:53.780 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.777+0000 7f9c1f7fe700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9c20101600 0x7f9c201964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:47:53.780 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.777+0000 7f9c1effd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c20103f30 0x7f9c201969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:47:53.780 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.778+0000 7f9c1effd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c20103f30 0x7f9c201969f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.22:37314/0 (socket says 172.21.15.22:37314) 2024-03-24T09:47:53.780 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.778+0000 7f9c1effd700 1 -- 172.21.15.22:0/2897230369 learned_addr learned my addr 172.21.15.22:0/2897230369 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:47:53.780 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.778+0000 7f9c1effd700 1 -- 172.21.15.22:0/2897230369 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9c20101600 msgr2=0x7f9c201964b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:53.780 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.778+0000 7f9c1effd700 1 --2- 172.21.15.22:0/2897230369 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9c20101600 0x7f9c201964b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:53.780 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.778+0000 7f9c1effd700 1 -- 172.21.15.22:0/2897230369 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c140095f0 con 0x7f9c20103f30 2024-03-24T09:47:53.780 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.778+0000 7f9c1effd700 1 --2- 172.21.15.22:0/2897230369 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c20103f30 0x7f9c201969f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9c1400b800 tx=0x7f9c14005f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:47:53.780 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.778+0000 7f9c1cff9700 1 -- 172.21.15.22:0/2897230369 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c1401d070 con 0x7f9c20103f30 2024-03-24T09:47:53.780 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.778+0000 7f9c260e1700 1 -- 172.21.15.22:0/2897230369 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c2019bb50 con 0x7f9c20103f30 2024-03-24T09:47:53.780 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.778+0000 7f9c1cff9700 1 -- 172.21.15.22:0/2897230369 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9c140052a0 con 0x7f9c20103f30 2024-03-24T09:47:53.781 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.778+0000 7f9c1cff9700 1 -- 172.21.15.22:0/2897230369 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c14017400 con 0x7f9c20103f30 2024-03-24T09:47:53.781 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.778+0000 7f9c260e1700 1 -- 172.21.15.22:0/2897230369 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c2019c070 con 0x7f9c20103f30 2024-03-24T09:47:53.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.780+0000 7f9c260e1700 1 -- 172.21.15.22:0/2897230369 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c2004ea80 con 0x7f9c20103f30 2024-03-24T09:47:53.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.780+0000 7f9c1cff9700 1 -- 172.21.15.22:0/2897230369 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f9c14003780 con 0x7f9c20103f30 2024-03-24T09:47:53.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.780+0000 7f9c1cff9700 1 --2- 172.21.15.22:0/2897230369 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f9c0806cce0 0x7f9c0806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:47:53.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.780+0000 7f9c1cff9700 1 -- 172.21.15.22:0/2897230369 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f9c1408d700 con 0x7f9c20103f30 2024-03-24T09:47:53.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.780+0000 7f9c1f7fe700 1 --2- 172.21.15.22:0/2897230369 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f9c0806cce0 0x7f9c0806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:47:53.782 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.781+0000 7f9c1f7fe700 1 --2- 172.21.15.22:0/2897230369 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f9c0806cce0 0x7f9c0806f1a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9c10009c60 tx=0x7f9c10009260 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:47:53.786 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:53.786+0000 7f9c1cff9700 1 -- 172.21.15.22:0/2897230369 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9c1405b500 con 0x7f9c20103f30 2024-03-24T09:47:53.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:53 smithi155 ceph-mon[37016]: from='client.24323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:47:53.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:53 smithi155 ceph-mon[37016]: pgmap v221: 65 pgs: 65 active+clean; 601 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.6 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-03-24T09:47:53.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:53 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:47:53.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:53 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:47:53.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:53 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:47:53.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:53 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:47:54.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:53 smithi022 ceph-mon[29512]: from='client.24323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:47:54.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:53 smithi022 ceph-mon[29512]: pgmap v221: 65 pgs: 65 active+clean; 601 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.6 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-03-24T09:47:54.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:53 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:47:54.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:53 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:47:54.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:53 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' 2024-03-24T09:47:54.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:53 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:47:54.077 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:54.076+0000 7f9c260e1700 1 -- 172.21.15.22:0/2897230369 --> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9c20105dc0 con 0x7f9c0806cce0 2024-03-24T09:47:54.078 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:54.077+0000 7f9c1cff9700 1 -- 172.21.15.22:0/2897230369 <== mgr.14215 v2:172.21.15.22:6800/773863266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f9c20105dc0 con 0x7f9c0806cce0 2024-03-24T09:47:54.078 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T09:47:54.078 INFO:teuthology.orchestra.run.smithi022.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919", 2024-03-24T09:47:54.078 INFO:teuthology.orchestra.run.smithi022.stdout: "in_progress": true, 2024-03-24T09:47:54.078 INFO:teuthology.orchestra.run.smithi022.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-24T09:47:54.079 INFO:teuthology.orchestra.run.smithi022.stdout: "services_complete": [], 2024-03-24T09:47:54.079 INFO:teuthology.orchestra.run.smithi022.stdout: "progress": "0/2 daemons upgraded", 2024-03-24T09:47:54.079 INFO:teuthology.orchestra.run.smithi022.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 image on host smithi155", 2024-03-24T09:47:54.079 INFO:teuthology.orchestra.run.smithi022.stdout: "is_paused": false 2024-03-24T09:47:54.079 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T09:47:54.081 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:54.081+0000 7f9c260e1700 1 -- 172.21.15.22:0/2897230369 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f9c0806cce0 msgr2=0x7f9c0806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:54.081 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:54.081+0000 7f9c260e1700 1 --2- 172.21.15.22:0/2897230369 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f9c0806cce0 0x7f9c0806f1a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9c10009c60 tx=0x7f9c10009260 comp rx=0 tx=0).stop 2024-03-24T09:47:54.081 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:54.081+0000 7f9c260e1700 1 -- 172.21.15.22:0/2897230369 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c20103f30 msgr2=0x7f9c201969f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:47:54.081 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:54.081+0000 7f9c260e1700 1 --2- 172.21.15.22:0/2897230369 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c20103f30 0x7f9c201969f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9c1400b800 tx=0x7f9c14005f00 comp rx=0 tx=0).stop 2024-03-24T09:47:54.082 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:54.081+0000 7f9c260e1700 1 -- 172.21.15.22:0/2897230369 shutdown_connections 2024-03-24T09:47:54.082 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:54.081+0000 7f9c260e1700 1 --2- 172.21.15.22:0/2897230369 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f9c0806cce0 0x7f9c0806f1a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:54.082 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:54.081+0000 7f9c260e1700 1 --2- 172.21.15.22:0/2897230369 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9c20101600 0x7f9c201964b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:54.082 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:54.081+0000 7f9c260e1700 1 --2- 172.21.15.22:0/2897230369 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c20103f30 0x7f9c201969f0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:47:54.082 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:54.081+0000 7f9c260e1700 1 -- 172.21.15.22:0/2897230369 >> 172.21.15.22:0/2897230369 conn(0x7f9c200faf70 msgr2=0x7f9c200fd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:47:54.082 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:54.081+0000 7f9c260e1700 1 -- 172.21.15.22:0/2897230369 shutdown_connections 2024-03-24T09:47:54.082 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:47:54.081+0000 7f9c260e1700 1 -- 172.21.15.22:0/2897230369 wait complete. 2024-03-24T09:47:54.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:54 smithi155 ceph-mon[37016]: from='client.14570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:47:54.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:54 smithi155 ceph-mon[37016]: Upgrade: Target is version 19.0.0-1855-g21b3bd67 (unknown) 2024-03-24T09:47:54.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:54 smithi155 ceph-mon[37016]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, digests ['quay.ceph.io/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919'] 2024-03-24T09:47:54.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:54 smithi155 ceph-mon[37016]: Upgrade: Need to upgrade myself (mgr.smithi022.emgmxz) 2024-03-24T09:47:54.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:54 smithi155 ceph-mon[37016]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:47:54.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:54 smithi155 ceph-mon[37016]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 on smithi155 2024-03-24T09:47:54.962 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:54 smithi155 ceph-mon[37016]: from='client.? 172.21.15.22:0/324992418' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T09:47:55.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:54 smithi022 ceph-mon[29512]: from='client.14570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:47:55.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:54 smithi022 ceph-mon[29512]: Upgrade: Target is version 19.0.0-1855-g21b3bd67 (unknown) 2024-03-24T09:47:55.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:54 smithi022 ceph-mon[29512]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, digests ['quay.ceph.io/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919'] 2024-03-24T09:47:55.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:54 smithi022 ceph-mon[29512]: Upgrade: Need to upgrade myself (mgr.smithi022.emgmxz) 2024-03-24T09:47:55.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:54 smithi022 ceph-mon[29512]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:47:55.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:54 smithi022 ceph-mon[29512]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 on smithi155 2024-03-24T09:47:55.033 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:54 smithi022 ceph-mon[29512]: from='client.? 172.21.15.22:0/324992418' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T09:47:55.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:55 smithi155 ceph-mon[37016]: from='client.24339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:47:55.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:55 smithi155 ceph-mon[37016]: pgmap v222: 65 pgs: 65 active+clean; 3.5 MiB data, 172 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 259 KiB/s wr, 2 op/s 2024-03-24T09:47:56.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:55 smithi022 ceph-mon[29512]: from='client.24339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:47:56.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:55 smithi022 ceph-mon[29512]: pgmap v222: 65 pgs: 65 active+clean; 3.5 MiB data, 172 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 259 KiB/s wr, 2 op/s 2024-03-24T09:47:57.194 INFO:tasks.workunit.client.0.smithi022.stderr:Note: switching to 'd39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b'. 2024-03-24T09:47:57.195 INFO:tasks.workunit.client.0.smithi022.stderr: 2024-03-24T09:47:57.195 INFO:tasks.workunit.client.0.smithi022.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-24T09:47:57.195 INFO:tasks.workunit.client.0.smithi022.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-24T09:47:57.195 INFO:tasks.workunit.client.0.smithi022.stderr:state without impacting any branches by switching back to a branch. 2024-03-24T09:47:57.195 INFO:tasks.workunit.client.0.smithi022.stderr: 2024-03-24T09:47:57.195 INFO:tasks.workunit.client.0.smithi022.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-24T09:47:57.195 INFO:tasks.workunit.client.0.smithi022.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-24T09:47:57.195 INFO:tasks.workunit.client.0.smithi022.stderr: 2024-03-24T09:47:57.196 INFO:tasks.workunit.client.0.smithi022.stderr: git switch -c 2024-03-24T09:47:57.196 INFO:tasks.workunit.client.0.smithi022.stderr: 2024-03-24T09:47:57.196 INFO:tasks.workunit.client.0.smithi022.stderr:Or undo this operation with: 2024-03-24T09:47:57.196 INFO:tasks.workunit.client.0.smithi022.stderr: 2024-03-24T09:47:57.196 INFO:tasks.workunit.client.0.smithi022.stderr: git switch - 2024-03-24T09:47:57.196 INFO:tasks.workunit.client.0.smithi022.stderr: 2024-03-24T09:47:57.196 INFO:tasks.workunit.client.0.smithi022.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-24T09:47:57.196 INFO:tasks.workunit.client.0.smithi022.stderr: 2024-03-24T09:47:57.196 INFO:tasks.workunit.client.0.smithi022.stderr:HEAD is now at d39ab7bb1f1 qa/suites/fs/workload: enable snap_schedule early 2024-03-24T09:47:57.199 DEBUG:teuthology.orchestra.run.smithi022:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-24T09:47:57.223 INFO:tasks.workunit.client.0.smithi022.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-24T09:47:57.225 INFO:tasks.workunit.client.0.smithi022.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-24T09:47:57.225 INFO:tasks.workunit.client.0.smithi022.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-24T09:47:57.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:57 smithi155 ceph-mon[37016]: pgmap v223: 65 pgs: 65 active+clean; 14 MiB data, 190 MiB used, 536 GiB / 536 GiB avail; 342 KiB/s rd, 1.1 MiB/s wr, 66 op/s 2024-03-24T09:47:58.031 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:57 smithi022 ceph-mon[29512]: pgmap v223: 65 pgs: 65 active+clean; 14 MiB data, 190 MiB used, 536 GiB / 536 GiB avail; 342 KiB/s rd, 1.1 MiB/s wr, 66 op/s 2024-03-24T09:47:58.516 INFO:tasks.workunit.client.0.smithi022.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-24T09:47:58.578 INFO:tasks.workunit.client.0.smithi022.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-24T09:47:58.617 INFO:tasks.workunit.client.0.smithi022.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-24T09:47:58.618 INFO:tasks.workunit.client.0.smithi022.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-24T09:47:58.618 INFO:tasks.workunit.client.0.smithi022.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-24T09:47:58.656 INFO:tasks.workunit.client.0.smithi022.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-24T09:47:58.661 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T09:47:58.661 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-24T09:47:58.719 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-03-24T09:47:58.720 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-03-24T09:47:58.720 DEBUG:teuthology.orchestra.run.smithi022:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-03-24T09:47:58.782 INFO:tasks.workunit.client.0.smithi022.stderr:+ mkdir -p fsstress 2024-03-24T09:47:58.784 INFO:tasks.workunit.client.0.smithi022.stderr:+ pushd fsstress 2024-03-24T09:47:58.784 INFO:tasks.workunit.client.0.smithi022.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-03-24T09:47:58.784 INFO:tasks.workunit.client.0.smithi022.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-03-24T09:47:59.570 INFO:tasks.workunit.client.0.smithi022.stderr:+ tar xzf ltp-full.tgz 2024-03-24T09:47:59.853 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:47:59 smithi022 ceph-mon[29512]: pgmap v224: 65 pgs: 65 active+clean; 31 MiB data, 279 MiB used, 536 GiB / 536 GiB avail; 683 KiB/s rd, 2.7 MiB/s wr, 222 op/s 2024-03-24T09:47:59.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:47:59 smithi155 ceph-mon[37016]: pgmap v224: 65 pgs: 65 active+clean; 31 MiB data, 279 MiB used, 536 GiB / 536 GiB avail; 683 KiB/s rd, 2.7 MiB/s wr, 222 op/s 2024-03-24T09:48:01.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:48:01 smithi155 ceph-mon[37016]: pgmap v225: 65 pgs: 65 active+clean; 49 MiB data, 341 MiB used, 536 GiB / 536 GiB avail; 1.0 MiB/s rd, 4.2 MiB/s wr, 396 op/s 2024-03-24T09:48:01.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:48:01 smithi022 ceph-mon[29512]: pgmap v225: 65 pgs: 65 active+clean; 49 MiB data, 341 MiB used, 536 GiB / 536 GiB avail; 1.0 MiB/s rd, 4.2 MiB/s wr, 396 op/s 2024-03-24T09:48:03.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:48:03 smithi155 ceph-mon[37016]: pgmap v226: 65 pgs: 65 active+clean; 54 MiB data, 366 MiB used, 536 GiB / 536 GiB avail; 1.7 MiB/s rd, 4.7 MiB/s wr, 408 op/s 2024-03-24T09:48:04.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:48:03 smithi022 ceph-mon[29512]: pgmap v226: 65 pgs: 65 active+clean; 54 MiB data, 366 MiB used, 536 GiB / 536 GiB avail; 1.7 MiB/s rd, 4.7 MiB/s wr, 408 op/s 2024-03-24T09:48:05.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:48:05 smithi155 ceph-mon[37016]: pgmap v227: 65 pgs: 65 active+clean; 58 MiB data, 386 MiB used, 536 GiB / 536 GiB avail; 2.0 MiB/s rd, 5.1 MiB/s wr, 411 op/s 2024-03-24T09:48:06.031 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:48:05 smithi022 ceph-mon[29512]: pgmap v227: 65 pgs: 65 active+clean; 58 MiB data, 386 MiB used, 536 GiB / 536 GiB avail; 2.0 MiB/s rd, 5.1 MiB/s wr, 411 op/s 2024-03-24T09:48:07.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:48:07 smithi155 ceph-mon[37016]: pgmap v228: 65 pgs: 65 active+clean; 79 MiB data, 433 MiB used, 536 GiB / 536 GiB avail; 2.7 MiB/s rd, 6.6 MiB/s wr, 489 op/s 2024-03-24T09:48:08.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:48:07 smithi022 ceph-mon[29512]: pgmap v228: 65 pgs: 65 active+clean; 79 MiB data, 433 MiB used, 536 GiB / 536 GiB avail; 2.7 MiB/s rd, 6.6 MiB/s wr, 489 op/s 2024-03-24T09:48:08.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:48:08 smithi155 ceph-mon[37016]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:48:09.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:48:08 smithi022 ceph-mon[29512]: from='mgr.14215 172.21.15.22:0/2375188415' entity='mgr.smithi022.emgmxz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T09:48:09.854 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:48:09 smithi022 ceph-mon[29512]: pgmap v229: 65 pgs: 65 active+clean; 82 MiB data, 480 MiB used, 536 GiB / 536 GiB avail; 2.3 MiB/s rd, 6.0 MiB/s wr, 490 op/s 2024-03-24T09:48:09.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:48:09 smithi155 ceph-mon[37016]: pgmap v229: 65 pgs: 65 active+clean; 82 MiB data, 480 MiB used, 536 GiB / 536 GiB avail; 2.3 MiB/s rd, 6.0 MiB/s wr, 490 op/s 2024-03-24T09:48:11.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:48:11 smithi155 ceph-mon[37016]: pgmap v230: 65 pgs: 65 active+clean; 99 MiB data, 517 MiB used, 536 GiB / 536 GiB avail; 2.5 MiB/s rd, 5.9 MiB/s wr, 460 op/s 2024-03-24T09:48:11.531 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:48:11 smithi022 ceph-mon[29512]: pgmap v230: 65 pgs: 65 active+clean; 99 MiB data, 517 MiB used, 536 GiB / 536 GiB avail; 2.5 MiB/s rd, 5.9 MiB/s wr, 460 op/s 2024-03-24T09:48:13.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:48:13 smithi155 ceph-mon[37016]: pgmap v231: 65 pgs: 65 active+clean; 105 MiB data, 537 MiB used, 536 GiB / 536 GiB avail; 2.7 MiB/s rd, 4.9 MiB/s wr, 354 op/s 2024-03-24T09:48:14.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:48:13 smithi022 ceph-mon[29512]: pgmap v231: 65 pgs: 65 active+clean; 105 MiB data, 537 MiB used, 536 GiB / 536 GiB avail; 2.7 MiB/s rd, 4.9 MiB/s wr, 354 op/s 2024-03-24T09:48:15.834 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:48:15 smithi022 ceph-mon[29512]: pgmap v232: 65 pgs: 65 active+clean; 119 MiB data, 616 MiB used, 536 GiB / 536 GiB avail; 2.4 MiB/s rd, 5.7 MiB/s wr, 571 op/s 2024-03-24T09:48:15.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:48:15 smithi155 ceph-mon[37016]: pgmap v232: 65 pgs: 65 active+clean; 119 MiB data, 616 MiB used, 536 GiB / 536 GiB avail; 2.4 MiB/s rd, 5.7 MiB/s wr, 571 op/s 2024-03-24T09:48:17.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:48:17 smithi155 ceph-mon[37016]: pgmap v233: 65 pgs: 65 active+clean; 135 MiB data, 663 MiB used, 536 GiB / 536 GiB avail; 2.6 MiB/s rd, 6.8 MiB/s wr, 679 op/s 2024-03-24T09:48:18.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:48:17 smithi022 ceph-mon[29512]: pgmap v233: 65 pgs: 65 active+clean; 135 MiB data, 663 MiB used, 536 GiB / 536 GiB avail; 2.6 MiB/s rd, 6.8 MiB/s wr, 679 op/s 2024-03-24T09:48:19.854 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:48:19 smithi022 ceph-mon[29512]: pgmap v234: 65 pgs: 65 active+clean; 145 MiB data, 725 MiB used, 536 GiB / 536 GiB avail; 2.2 MiB/s rd, 5.9 MiB/s wr, 716 op/s 2024-03-24T09:48:19.961 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:48:19 smithi155 ceph-mon[37016]: pgmap v234: 65 pgs: 65 active+clean; 145 MiB data, 725 MiB used, 536 GiB / 536 GiB avail; 2.2 MiB/s rd, 5.9 MiB/s wr, 716 op/s 2024-03-24T09:48:20.117 INFO:tasks.workunit.client.1.smithi155.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-03-24T09:48:20.126 INFO:tasks.workunit.client.1.smithi155.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-03-24T09:48:20.126 INFO:tasks.workunit.client.1.smithi155.stderr:+ make 2024-03-24T09:48:20.958 INFO:tasks.workunit.client.1.smithi155.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-03-24T09:48:21.461 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:Mar 24 09:48:21 smithi155 ceph-mon[37016]: pgmap v235: 65 pgs: 65 active+clean; 153 MiB data, 765 MiB used, 536 GiB / 536 GiB avail; 2.4 MiB/s rd, 6.3 MiB/s wr, 735 op/s 2024-03-24T09:48:21.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:48:21 smithi022 ceph-mon[29512]: pgmap v235: 65 pgs: 65 active+clean; 153 MiB data, 765 MiB used, 536 GiB / 536 GiB avail; 2.4 MiB/s rd, 6.3 MiB/s wr, 735 op/s 2024-03-24T09:48:24.208 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.206+0000 7f75e4bde700 1 -- 172.21.15.22:0/3271300032 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f75e0100ae0 msgr2=0x7f75e0100f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:48:24.209 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.206+0000 7f75e4bde700 1 --2- 172.21.15.22:0/3271300032 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f75e0100ae0 0x7f75e0100f00 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f75d0009910 tx=0x7f75d0009c20 comp rx=0 tx=0).stop 2024-03-24T09:48:24.209 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.206+0000 7f75e4bde700 1 -- 172.21.15.22:0/3271300032 shutdown_connections 2024-03-24T09:48:24.209 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.206+0000 7f75e4bde700 1 --2- 172.21.15.22:0/3271300032 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75e0101ce0 0x7f75e0102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:48:24.209 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.206+0000 7f75e4bde700 1 --2- 172.21.15.22:0/3271300032 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f75e0100ae0 0x7f75e0100f00 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:48:24.209 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.206+0000 7f75e4bde700 1 -- 172.21.15.22:0/3271300032 >> 172.21.15.22:0/3271300032 conn(0x7f75e00fc0a0 msgr2=0x7f75e00fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:48:24.209 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.206+0000 7f75e4bde700 1 -- 172.21.15.22:0/3271300032 shutdown_connections 2024-03-24T09:48:24.209 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.206+0000 7f75e4bde700 1 -- 172.21.15.22:0/3271300032 wait complete. 2024-03-24T09:48:24.209 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.207+0000 7f75e4bde700 1 Processor -- start 2024-03-24T09:48:24.209 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.207+0000 7f75e4bde700 1 -- start start 2024-03-24T09:48:24.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.207+0000 7f75e4bde700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f75e0100ae0 0x7f75e0198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:48:24.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.207+0000 7f75e4bde700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75e0101ce0 0x7f75e0198d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:48:24.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.207+0000 7f75e4bde700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e0199390 con 0x7f75e0100ae0 2024-03-24T09:48:24.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.207+0000 7f75e4bde700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e01994d0 con 0x7f75e0101ce0 2024-03-24T09:48:24.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.207+0000 7f75de59c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f75e0100ae0 0x7f75e0198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:48:24.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.207+0000 7f75de59c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f75e0100ae0 0x7f75e0198830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42314/0 (socket says 172.21.15.22:42314) 2024-03-24T09:48:24.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.207+0000 7f75de59c700 1 -- 172.21.15.22:0/2978012763 learned_addr learned my addr 172.21.15.22:0/2978012763 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:48:24.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.207+0000 7f75de59c700 1 -- 172.21.15.22:0/2978012763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75e0101ce0 msgr2=0x7f75e0198d70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-24T09:48:24.210 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.207+0000 7f75de59c700 1 --2- 172.21.15.22:0/2978012763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75e0101ce0 0x7f75e0198d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:48:24.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.207+0000 7f75de59c700 1 -- 172.21.15.22:0/2978012763 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d00095f0 con 0x7f75e0100ae0 2024-03-24T09:48:24.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.207+0000 7f75de59c700 1 --2- 172.21.15.22:0/2978012763 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f75e0100ae0 0x7f75e0198830 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f75d000bd70 tx=0x7f75d000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:48:24.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.208+0000 7f75cf7fe700 1 -- 172.21.15.22:0/2978012763 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75d001d070 con 0x7f75e0100ae0 2024-03-24T09:48:24.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.208+0000 7f75cf7fe700 1 -- 172.21.15.22:0/2978012763 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f75d000f460 con 0x7f75e0100ae0 2024-03-24T09:48:24.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.208+0000 7f75cf7fe700 1 -- 172.21.15.22:0/2978012763 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75d0021640 con 0x7f75e0100ae0 2024-03-24T09:48:24.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.208+0000 7f75e4bde700 1 -- 172.21.15.22:0/2978012763 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75e019ded0 con 0x7f75e0100ae0 2024-03-24T09:48:24.211 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.208+0000 7f75e4bde700 1 -- 172.21.15.22:0/2978012763 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75e019e3c0 con 0x7f75e0100ae0 2024-03-24T09:48:24.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.209+0000 7f75e4bde700 1 -- 172.21.15.22:0/2978012763 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75e01066b0 con 0x7f75e0100ae0 2024-03-24T09:48:24.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.209+0000 7f75cf7fe700 1 -- 172.21.15.22:0/2978012763 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f75d000fa90 con 0x7f75e0100ae0 2024-03-24T09:48:24.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.210+0000 7f75cf7fe700 1 --2- 172.21.15.22:0/2978012763 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f75c806cce0 0x7f75c806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:48:24.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.210+0000 7f75cf7fe700 1 -- 172.21.15.22:0/2978012763 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f75d008d5f0 con 0x7f75e0100ae0 2024-03-24T09:48:24.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.210+0000 7f75ddd9b700 1 --2- 172.21.15.22:0/2978012763 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f75c806cce0 0x7f75c806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:48:24.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.210+0000 7f75ddd9b700 1 --2- 172.21.15.22:0/2978012763 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f75c806cce0 0x7f75c806f1a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f75d4009710 tx=0x7f75d4006c60 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:48:24.215 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.215+0000 7f75cf7fe700 1 -- 172.21.15.22:0/2978012763 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f75d005b3f0 con 0x7f75e0100ae0 2024-03-24T09:48:24.434 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.434+0000 7f75e4bde700 1 -- 172.21.15.22:0/2978012763 --> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f75e00611f0 con 0x7f75c806cce0 2024-03-24T09:48:24.435 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.435+0000 7f75cf7fe700 1 -- 172.21.15.22:0/2978012763 <== mgr.14215 v2:172.21.15.22:6800/773863266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f75e00611f0 con 0x7f75c806cce0 2024-03-24T09:48:24.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.437+0000 7f75e4bde700 1 -- 172.21.15.22:0/2978012763 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f75c806cce0 msgr2=0x7f75c806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:48:24.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.437+0000 7f75e4bde700 1 --2- 172.21.15.22:0/2978012763 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f75c806cce0 0x7f75c806f1a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f75d4009710 tx=0x7f75d4006c60 comp rx=0 tx=0).stop 2024-03-24T09:48:24.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.437+0000 7f75e4bde700 1 -- 172.21.15.22:0/2978012763 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f75e0100ae0 msgr2=0x7f75e0198830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:48:24.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.437+0000 7f75e4bde700 1 --2- 172.21.15.22:0/2978012763 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f75e0100ae0 0x7f75e0198830 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f75d000bd70 tx=0x7f75d000be50 comp rx=0 tx=0).stop 2024-03-24T09:48:24.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.437+0000 7f75e4bde700 1 -- 172.21.15.22:0/2978012763 shutdown_connections 2024-03-24T09:48:24.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.437+0000 7f75e4bde700 1 --2- 172.21.15.22:0/2978012763 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f75c806cce0 0x7f75c806f1a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:48:24.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.437+0000 7f75e4bde700 1 --2- 172.21.15.22:0/2978012763 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f75e0100ae0 0x7f75e0198830 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:48:24.438 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.437+0000 7f75e4bde700 1 --2- 172.21.15.22:0/2978012763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75e0101ce0 0x7f75e0198d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:48:24.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.437+0000 7f75e4bde700 1 -- 172.21.15.22:0/2978012763 >> 172.21.15.22:0/2978012763 conn(0x7f75e00fc0a0 msgr2=0x7f75e0073400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:48:24.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.437+0000 7f75e4bde700 1 -- 172.21.15.22:0/2978012763 shutdown_connections 2024-03-24T09:48:24.439 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.437+0000 7f75e4bde700 1 -- 172.21.15.22:0/2978012763 wait complete. 2024-03-24T09:48:24.447 INFO:teuthology.orchestra.run.smithi022.stdout:true 2024-03-24T09:48:24.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.543+0000 7fde7589c700 1 -- 172.21.15.22:0/2103437612 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde70101cd0 msgr2=0x7fde70102130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:48:24.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.543+0000 7fde7589c700 1 --2- 172.21.15.22:0/2103437612 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde70101cd0 0x7fde70102130 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fde58009910 tx=0x7fde58009c20 comp rx=0 tx=0).stop 2024-03-24T09:48:24.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.543+0000 7fde7589c700 1 -- 172.21.15.22:0/2103437612 shutdown_connections 2024-03-24T09:48:24.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.543+0000 7fde7589c700 1 --2- 172.21.15.22:0/2103437612 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde70101cd0 0x7fde70102130 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:48:24.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.543+0000 7fde7589c700 1 --2- 172.21.15.22:0/2103437612 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde70100ad0 0x7fde70100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:48:24.545 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.543+0000 7fde7589c700 1 -- 172.21.15.22:0/2103437612 >> 172.21.15.22:0/2103437612 conn(0x7fde700fc090 msgr2=0x7fde700fe4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:48:24.546 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.543+0000 7fde7589c700 1 -- 172.21.15.22:0/2103437612 shutdown_connections 2024-03-24T09:48:24.546 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.543+0000 7fde7589c700 1 -- 172.21.15.22:0/2103437612 wait complete. 2024-03-24T09:48:24.546 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.544+0000 7fde7589c700 1 Processor -- start 2024-03-24T09:48:24.546 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.544+0000 7fde7589c700 1 -- start start 2024-03-24T09:48:24.546 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.544+0000 7fde7589c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde70100ad0 0x7fde70074a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:48:24.546 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.544+0000 7fde7589c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde70101cd0 0x7fde700731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:48:24.546 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.544+0000 7fde7589c700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde70075060 con 0x7fde70101cd0 2024-03-24T09:48:24.546 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.544+0000 7fde7589c700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde700736e0 con 0x7fde70100ad0 2024-03-24T09:48:24.546 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.544+0000 7fde6e7fc700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde70101cd0 0x7fde700731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:48:24.546 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.544+0000 7fde6e7fc700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde70101cd0 0x7fde700731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42336/0 (socket says 172.21.15.22:42336) 2024-03-24T09:48:24.546 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.544+0000 7fde6e7fc700 1 -- 172.21.15.22:0/4196047283 learned_addr learned my addr 172.21.15.22:0/4196047283 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:48:24.546 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.544+0000 7fde6e7fc700 1 -- 172.21.15.22:0/4196047283 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde70100ad0 msgr2=0x7fde70074a40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-24T09:48:24.546 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.544+0000 7fde6e7fc700 1 --2- 172.21.15.22:0/4196047283 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde70100ad0 0x7fde70074a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:48:24.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.544+0000 7fde6e7fc700 1 -- 172.21.15.22:0/4196047283 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde580095f0 con 0x7fde70101cd0 2024-03-24T09:48:24.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.544+0000 7fde6e7fc700 1 --2- 172.21.15.22:0/4196047283 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde70101cd0 0x7fde700731a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fde58005090 tx=0x7fde5800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:48:24.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.545+0000 7fde7489a700 1 -- 172.21.15.22:0/4196047283 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde5801d070 con 0x7fde70101cd0 2024-03-24T09:48:24.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.545+0000 7fde7489a700 1 -- 172.21.15.22:0/4196047283 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fde58022470 con 0x7fde70101cd0 2024-03-24T09:48:24.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.545+0000 7fde7589c700 1 -- 172.21.15.22:0/4196047283 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde70073910 con 0x7fde70101cd0 2024-03-24T09:48:24.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.545+0000 7fde7489a700 1 -- 172.21.15.22:0/4196047283 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde5800f690 con 0x7fde70101cd0 2024-03-24T09:48:24.547 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.545+0000 7fde7589c700 1 -- 172.21.15.22:0/4196047283 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde70073e80 con 0x7fde70101cd0 2024-03-24T09:48:24.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.546+0000 7fde7589c700 1 -- 172.21.15.22:0/4196047283 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde701906f0 con 0x7fde70101cd0 2024-03-24T09:48:24.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.546+0000 7fde7489a700 1 -- 172.21.15.22:0/4196047283 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7fde580225e0 con 0x7fde70101cd0 2024-03-24T09:48:24.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.546+0000 7fde7489a700 1 --2- 172.21.15.22:0/4196047283 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fde5c06cce0 0x7fde5c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:48:24.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.547+0000 7fde7489a700 1 -- 172.21.15.22:0/4196047283 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fde5808dba0 con 0x7fde70101cd0 2024-03-24T09:48:24.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.547+0000 7fde6effd700 1 --2- 172.21.15.22:0/4196047283 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fde5c06cce0 0x7fde5c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:48:24.548 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.547+0000 7fde6effd700 1 --2- 172.21.15.22:0/4196047283 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fde5c06cce0 0x7fde5c06f1a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fde70101b30 tx=0x7fde68006c60 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:48:24.551 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.551+0000 7fde7489a700 1 -- 172.21.15.22:0/4196047283 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fde5805bbf0 con 0x7fde70101cd0 2024-03-24T09:48:24.771 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.771+0000 7fde7589c700 1 -- 172.21.15.22:0/4196047283 --> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fde700611f0 con 0x7fde5c06cce0 2024-03-24T09:48:24.772 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.772+0000 7fde7489a700 1 -- 172.21.15.22:0/4196047283 <== mgr.14215 v2:172.21.15.22:6800/773863266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7fde700611f0 con 0x7fde5c06cce0 2024-03-24T09:48:24.774 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.774+0000 7fde7589c700 1 -- 172.21.15.22:0/4196047283 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fde5c06cce0 msgr2=0x7fde5c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:48:24.774 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.774+0000 7fde7589c700 1 --2- 172.21.15.22:0/4196047283 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fde5c06cce0 0x7fde5c06f1a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fde70101b30 tx=0x7fde68006c60 comp rx=0 tx=0).stop 2024-03-24T09:48:24.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.774+0000 7fde7589c700 1 -- 172.21.15.22:0/4196047283 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde70101cd0 msgr2=0x7fde700731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:48:24.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.774+0000 7fde7589c700 1 --2- 172.21.15.22:0/4196047283 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde70101cd0 0x7fde700731a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fde58005090 tx=0x7fde5800be20 comp rx=0 tx=0).stop 2024-03-24T09:48:24.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.774+0000 7fde7589c700 1 -- 172.21.15.22:0/4196047283 shutdown_connections 2024-03-24T09:48:24.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.774+0000 7fde7589c700 1 --2- 172.21.15.22:0/4196047283 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fde5c06cce0 0x7fde5c06f1a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:48:24.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.774+0000 7fde7589c700 1 --2- 172.21.15.22:0/4196047283 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde70100ad0 0x7fde70074a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:48:24.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.774+0000 7fde7589c700 1 --2- 172.21.15.22:0/4196047283 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde70101cd0 0x7fde700731a0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:48:24.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.774+0000 7fde7589c700 1 -- 172.21.15.22:0/4196047283 >> 172.21.15.22:0/4196047283 conn(0x7fde700fc090 msgr2=0x7fde70105320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:48:24.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.774+0000 7fde7589c700 1 -- 172.21.15.22:0/4196047283 shutdown_connections 2024-03-24T09:48:24.775 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.774+0000 7fde7589c700 1 -- 172.21.15.22:0/4196047283 wait complete. 2024-03-24T09:48:24.880 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.878+0000 7fb774290700 1 -- 172.21.15.22:0/1228397903 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb76c101cd0 msgr2=0x7fb76c102130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:48:24.880 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.878+0000 7fb774290700 1 --2- 172.21.15.22:0/1228397903 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb76c101cd0 0x7fb76c102130 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb768009910 tx=0x7fb768009c20 comp rx=0 tx=0).stop 2024-03-24T09:48:24.880 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.878+0000 7fb774290700 1 -- 172.21.15.22:0/1228397903 shutdown_connections 2024-03-24T09:48:24.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.878+0000 7fb774290700 1 --2- 172.21.15.22:0/1228397903 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb76c101cd0 0x7fb76c102130 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:48:24.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.878+0000 7fb774290700 1 --2- 172.21.15.22:0/1228397903 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb76c100ad0 0x7fb76c100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:48:24.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.878+0000 7fb774290700 1 -- 172.21.15.22:0/1228397903 >> 172.21.15.22:0/1228397903 conn(0x7fb76c0fc070 msgr2=0x7fb76c0fe4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:48:24.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.878+0000 7fb774290700 1 -- 172.21.15.22:0/1228397903 shutdown_connections 2024-03-24T09:48:24.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.878+0000 7fb774290700 1 -- 172.21.15.22:0/1228397903 wait complete. 2024-03-24T09:48:24.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.879+0000 7fb774290700 1 Processor -- start 2024-03-24T09:48:24.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.879+0000 7fb774290700 1 -- start start 2024-03-24T09:48:24.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.879+0000 7fb774290700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb76c100ad0 0x7fb76c1965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:48:24.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.879+0000 7fb774290700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb76c101cd0 0x7fb76c196af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:48:24.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.879+0000 7fb774290700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76c197110 con 0x7fb76c101cd0 2024-03-24T09:48:24.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.879+0000 7fb774290700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76c197250 con 0x7fb76c100ad0 2024-03-24T09:48:24.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.879+0000 7fb77182b700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb76c101cd0 0x7fb76c196af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:48:24.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.879+0000 7fb77182b700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb76c101cd0 0x7fb76c196af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42362/0 (socket says 172.21.15.22:42362) 2024-03-24T09:48:24.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.879+0000 7fb77182b700 1 -- 172.21.15.22:0/4275169477 learned_addr learned my addr 172.21.15.22:0/4275169477 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:48:24.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.879+0000 7fb77182b700 1 -- 172.21.15.22:0/4275169477 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb76c100ad0 msgr2=0x7fb76c1965b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-24T09:48:24.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.880+0000 7fb77182b700 1 --2- 172.21.15.22:0/4275169477 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb76c100ad0 0x7fb76c1965b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:48:24.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.880+0000 7fb77182b700 1 -- 172.21.15.22:0/4275169477 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7680095f0 con 0x7fb76c101cd0 2024-03-24T09:48:24.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.880+0000 7fb77182b700 1 --2- 172.21.15.22:0/4275169477 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb76c101cd0 0x7fb76c196af0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb768005090 tx=0x7fb76800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:48:24.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.880+0000 7fb75f7fe700 1 -- 172.21.15.22:0/4275169477 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb76801d070 con 0x7fb76c101cd0 2024-03-24T09:48:24.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.880+0000 7fb75f7fe700 1 -- 172.21.15.22:0/4275169477 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb768022470 con 0x7fb76c101cd0 2024-03-24T09:48:24.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.880+0000 7fb774290700 1 -- 172.21.15.22:0/4275169477 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb76c19bc50 con 0x7fb76c101cd0 2024-03-24T09:48:24.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.880+0000 7fb75f7fe700 1 -- 172.21.15.22:0/4275169477 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb76800f690 con 0x7fb76c101cd0 2024-03-24T09:48:24.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.880+0000 7fb774290700 1 -- 172.21.15.22:0/4275169477 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb76c19c1c0 con 0x7fb76c101cd0 2024-03-24T09:48:24.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.881+0000 7fb774290700 1 -- 172.21.15.22:0/4275169477 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb76c1906f0 con 0x7fb76c101cd0 2024-03-24T09:48:24.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.882+0000 7fb75f7fe700 1 -- 172.21.15.22:0/4275169477 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7fb7680225e0 con 0x7fb76c101cd0 2024-03-24T09:48:24.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.882+0000 7fb75f7fe700 1 --2- 172.21.15.22:0/4275169477 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fb75806cce0 0x7fb75806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:48:24.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.882+0000 7fb75f7fe700 1 -- 172.21.15.22:0/4275169477 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fb76808dba0 con 0x7fb76c101cd0 2024-03-24T09:48:24.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.882+0000 7fb77202c700 1 --2- 172.21.15.22:0/4275169477 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fb75806cce0 0x7fb75806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:48:24.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.882+0000 7fb77202c700 1 --2- 172.21.15.22:0/4275169477 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fb75806cce0 0x7fb75806f1a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb76c101b30 tx=0x7fb760006c60 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:48:24.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:24.887+0000 7fb75f7fe700 1 -- 172.21.15.22:0/4275169477 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb76805bbf0 con 0x7fb76c101cd0 2024-03-24T09:48:25.096 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.096+0000 7fb774290700 1 -- 172.21.15.22:0/4275169477 --> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb76c0611f0 con 0x7fb75806cce0 2024-03-24T09:48:25.107 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.106+0000 7fb75f7fe700 1 -- 172.21.15.22:0/4275169477 <== mgr.14215 v2:172.21.15.22:6800/773863266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7fb76c0611f0 con 0x7fb75806cce0 2024-03-24T09:48:25.107 INFO:teuthology.orchestra.run.smithi022.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T09:48:25.107 INFO:teuthology.orchestra.run.smithi022.stdout:alertmanager.smithi022 smithi022 *:9093,9094 running (5m) 84s ago 7m 25.1M - 0.25.0 c8568f914cd2 961b9fd08390 2024-03-24T09:48:25.107 INFO:teuthology.orchestra.run.smithi022.stdout:ceph-exporter.smithi022 smithi022 running (7m) 84s ago 7m 7457k - 18.2.2-952-g738c2fd6 68aff9aff3b2 513e822989e8 2024-03-24T09:48:25.108 INFO:teuthology.orchestra.run.smithi022.stdout:ceph-exporter.smithi155 smithi155 running (6m) 90s ago 6m 7390k - 18.2.2-952-g738c2fd6 68aff9aff3b2 2bc65c464d99 2024-03-24T09:48:25.108 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi022 smithi022 running (7m) 84s ago 7m 6665k - 18.2.2-952-g738c2fd6 68aff9aff3b2 a2a36122163c 2024-03-24T09:48:25.108 INFO:teuthology.orchestra.run.smithi022.stdout:crash.smithi155 smithi155 running (6m) 90s ago 6m 6665k - 18.2.2-952-g738c2fd6 68aff9aff3b2 2d0b688c3a61 2024-03-24T09:48:25.108 INFO:teuthology.orchestra.run.smithi022.stdout:grafana.smithi022 smithi022 *:3000 running (5m) 84s ago 7m 85.9M - 9.4.7 954c08fa6188 082a79d8dba0 2024-03-24T09:48:25.108 INFO:teuthology.orchestra.run.smithi022.stdout:mds.cephfs.smithi022.mhcbrm smithi022 running (111s) 84s ago 111s 11.3M - 18.2.2-952-g738c2fd6 68aff9aff3b2 e64e89f0270a 2024-03-24T09:48:25.108 INFO:teuthology.orchestra.run.smithi022.stdout:mds.cephfs.smithi022.qbmomh smithi022 running (104s) 84s ago 104s 17.0M - 18.2.2-952-g738c2fd6 68aff9aff3b2 77e763d4929f 2024-03-24T09:48:25.109 INFO:teuthology.orchestra.run.smithi022.stdout:mds.cephfs.smithi155.dakojr smithi155 running (102s) 90s ago 101s 13.1M - 18.2.2-952-g738c2fd6 68aff9aff3b2 279a6ccdd766 2024-03-24T09:48:25.109 INFO:teuthology.orchestra.run.smithi022.stdout:mds.cephfs.smithi155.pwhqcw smithi155 running (108s) 90s ago 108s 16.0M - 18.2.2-952-g738c2fd6 68aff9aff3b2 a3b6d24090c5 2024-03-24T09:48:25.109 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi022.emgmxz smithi022 *:9283,8765,8443 running (8m) 84s ago 8m 496M - 18.2.2-952-g738c2fd6 68aff9aff3b2 9c46d65fb5c7 2024-03-24T09:48:25.109 INFO:teuthology.orchestra.run.smithi022.stdout:mgr.smithi155.itwzew smithi155 *:8443,9283,8765 running (6m) 90s ago 6m 452M - 18.2.2-952-g738c2fd6 68aff9aff3b2 076d66e31d28 2024-03-24T09:48:25.109 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi022 smithi022 running (8m) 84s ago 8m 45.8M 2048M 18.2.2-952-g738c2fd6 68aff9aff3b2 59736938121b 2024-03-24T09:48:25.109 INFO:teuthology.orchestra.run.smithi022.stdout:mon.smithi155 smithi155 running (6m) 90s ago 6m 35.8M 2048M 18.2.2-952-g738c2fd6 68aff9aff3b2 9188cfee0421 2024-03-24T09:48:25.109 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi022 smithi022 *:9100 running (7m) 84s ago 7m 15.5M - 1.5.0 0da6a335fe13 0734bfdc392d 2024-03-24T09:48:25.110 INFO:teuthology.orchestra.run.smithi022.stdout:node-exporter.smithi155 smithi155 *:9100 running (6m) 90s ago 6m 14.5M - 1.5.0 0da6a335fe13 18110dca979b 2024-03-24T09:48:25.110 INFO:teuthology.orchestra.run.smithi022.stdout:osd.0 smithi022 running (5m) 84s ago 5m 46.9M 3948M 18.2.2-952-g738c2fd6 68aff9aff3b2 db726da62a1c 2024-03-24T09:48:25.110 INFO:teuthology.orchestra.run.smithi022.stdout:osd.1 smithi022 running (5m) 84s ago 5m 48.5M 3948M 18.2.2-952-g738c2fd6 68aff9aff3b2 3c6af2f5b6f1 2024-03-24T09:48:25.110 INFO:teuthology.orchestra.run.smithi022.stdout:osd.2 smithi022 running (4m) 84s ago 4m 46.6M 3948M 18.2.2-952-g738c2fd6 68aff9aff3b2 99b3c8840e7d 2024-03-24T09:48:25.110 INFO:teuthology.orchestra.run.smithi022.stdout:osd.3 smithi155 running (4m) 90s ago 4m 45.6M 4970M 18.2.2-952-g738c2fd6 68aff9aff3b2 534b4e3f3f15 2024-03-24T09:48:25.110 INFO:teuthology.orchestra.run.smithi022.stdout:osd.4 smithi155 running (3m) 90s ago 3m 43.9M 4970M 18.2.2-952-g738c2fd6 68aff9aff3b2 a269f005acda 2024-03-24T09:48:25.110 INFO:teuthology.orchestra.run.smithi022.stdout:osd.5 smithi155 running (3m) 90s ago 3m 42.8M 4970M 18.2.2-952-g738c2fd6 68aff9aff3b2 20dd2513a0df 2024-03-24T09:48:25.110 INFO:teuthology.orchestra.run.smithi022.stdout:prometheus.smithi022 smithi022 *:9095 running (5m) 84s ago 6m 55.1M - 2.43.0 a07b618ecd1d 5cb914be4e25 2024-03-24T09:48:25.111 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.108+0000 7fb774290700 1 -- 172.21.15.22:0/4275169477 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fb75806cce0 msgr2=0x7fb75806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:48:25.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.108+0000 7fb774290700 1 --2- 172.21.15.22:0/4275169477 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fb75806cce0 0x7fb75806f1a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb76c101b30 tx=0x7fb760006c60 comp rx=0 tx=0).stop 2024-03-24T09:48:25.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.108+0000 7fb774290700 1 -- 172.21.15.22:0/4275169477 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb76c101cd0 msgr2=0x7fb76c196af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:48:25.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.108+0000 7fb774290700 1 --2- 172.21.15.22:0/4275169477 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb76c101cd0 0x7fb76c196af0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb768005090 tx=0x7fb76800be20 comp rx=0 tx=0).stop 2024-03-24T09:48:25.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.108+0000 7fb774290700 1 -- 172.21.15.22:0/4275169477 shutdown_connections 2024-03-24T09:48:25.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.108+0000 7fb774290700 1 --2- 172.21.15.22:0/4275169477 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7fb75806cce0 0x7fb75806f1a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:48:25.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.108+0000 7fb774290700 1 --2- 172.21.15.22:0/4275169477 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb76c100ad0 0x7fb76c1965b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:48:25.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.108+0000 7fb774290700 1 --2- 172.21.15.22:0/4275169477 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb76c101cd0 0x7fb76c196af0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:48:25.112 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.108+0000 7fb774290700 1 -- 172.21.15.22:0/4275169477 >> 172.21.15.22:0/4275169477 conn(0x7fb76c0fc070 msgr2=0x7fb76c1052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:48:25.113 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.109+0000 7fb774290700 1 -- 172.21.15.22:0/4275169477 shutdown_connections 2024-03-24T09:48:25.113 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.109+0000 7fb774290700 1 -- 172.21.15.22:0/4275169477 wait complete. 2024-03-24T09:48:25.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.212+0000 7f3aaf780700 1 -- 172.21.15.22:0/360649241 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3aa80ff9e0 msgr2=0x7f3aa80ffe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:48:25.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.212+0000 7f3aaf780700 1 --2- 172.21.15.22:0/360649241 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3aa80ff9e0 0x7f3aa80ffe00 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f3a9800b0a0 tx=0x7f3a9800b3b0 comp rx=0 tx=0).stop 2024-03-24T09:48:25.212 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.212+0000 7f3aaf780700 1 -- 172.21.15.22:0/360649241 shutdown_connections 2024-03-24T09:48:25.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.212+0000 7f3aaf780700 1 --2- 172.21.15.22:0/360649241 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3aa8100be0 0x7f3aa8101040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:48:25.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.212+0000 7f3aaf780700 1 --2- 172.21.15.22:0/360649241 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3aa80ff9e0 0x7f3aa80ffe00 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:48:25.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.212+0000 7f3aaf780700 1 -- 172.21.15.22:0/360649241 >> 172.21.15.22:0/360649241 conn(0x7f3aa80faf80 msgr2=0x7f3aa80fd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:48:25.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.212+0000 7f3aaf780700 1 -- 172.21.15.22:0/360649241 shutdown_connections 2024-03-24T09:48:25.213 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.212+0000 7f3aaf780700 1 -- 172.21.15.22:0/360649241 wait complete. 2024-03-24T09:48:25.216 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.213+0000 7f3aaf780700 1 Processor -- start 2024-03-24T09:48:25.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.213+0000 7f3aaf780700 1 -- start start 2024-03-24T09:48:25.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.213+0000 7f3aaf780700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3aa80ff9e0 0x7f3aa8069af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:48:25.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.213+0000 7f3aaf780700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3aa8100be0 0x7f3aa819b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:48:25.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.213+0000 7f3aaf780700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa819b8f0 con 0x7f3aa80ff9e0 2024-03-24T09:48:25.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.213+0000 7f3aaf780700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa819ba30 con 0x7f3aa8100be0 2024-03-24T09:48:25.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.213+0000 7f3aad51c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3aa80ff9e0 0x7f3aa8069af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:48:25.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.213+0000 7f3aad51c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3aa80ff9e0 0x7f3aa8069af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42382/0 (socket says 172.21.15.22:42382) 2024-03-24T09:48:25.217 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.213+0000 7f3aad51c700 1 -- 172.21.15.22:0/787535248 learned_addr learned my addr 172.21.15.22:0/787535248 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:48:25.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.213+0000 7f3aad51c700 1 -- 172.21.15.22:0/787535248 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3aa8100be0 msgr2=0x7f3aa819b3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-24T09:48:25.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.213+0000 7f3aad51c700 1 --2- 172.21.15.22:0/787535248 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3aa8100be0 0x7f3aa819b3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:48:25.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.213+0000 7f3aad51c700 1 -- 172.21.15.22:0/787535248 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a98009d00 con 0x7f3aa80ff9e0 2024-03-24T09:48:25.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.213+0000 7f3aad51c700 1 --2- 172.21.15.22:0/787535248 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3aa80ff9e0 0x7f3aa8069af0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f3a98006130 tx=0x7f3a98003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:48:25.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.214+0000 7f3aa67fc700 1 -- 172.21.15.22:0/787535248 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a9800e040 con 0x7f3aa80ff9e0 2024-03-24T09:48:25.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.214+0000 7f3aa67fc700 1 -- 172.21.15.22:0/787535248 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3a980075c0 con 0x7f3aa80ff9e0 2024-03-24T09:48:25.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.214+0000 7f3aaf780700 1 -- 172.21.15.22:0/787535248 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aa819bc80 con 0x7f3aa80ff9e0 2024-03-24T09:48:25.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.214+0000 7f3aa67fc700 1 -- 172.21.15.22:0/787535248 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a9801dab0 con 0x7f3aa80ff9e0 2024-03-24T09:48:25.218 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.214+0000 7f3aaf780700 1 -- 172.21.15.22:0/787535248 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aa819c170 con 0x7f3aa80ff9e0 2024-03-24T09:48:25.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.215+0000 7f3aaf780700 1 -- 172.21.15.22:0/787535248 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3aa818f600 con 0x7f3aa80ff9e0 2024-03-24T09:48:25.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.215+0000 7f3aa67fc700 1 -- 172.21.15.22:0/787535248 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f3a9801dd30 con 0x7f3aa80ff9e0 2024-03-24T09:48:25.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.215+0000 7f3aa67fc700 1 --2- 172.21.15.22:0/787535248 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f3a9006cce0 0x7f3a9006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:48:25.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.216+0000 7f3aa67fc700 1 -- 172.21.15.22:0/787535248 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f3a9808dd60 con 0x7f3aa80ff9e0 2024-03-24T09:48:25.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.216+0000 7f3aacd1b700 1 --2- 172.21.15.22:0/787535248 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f3a9006cce0 0x7f3a9006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:48:25.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.216+0000 7f3aacd1b700 1 --2- 172.21.15.22:0/787535248 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f3a9006cce0 0x7f3a9006f1a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c009710 tx=0x7f3a9c006c60 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:48:25.220 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.220+0000 7f3aa67fc700 1 -- 172.21.15.22:0/787535248 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3a98091030 con 0x7f3aa80ff9e0 2024-03-24T09:48:25.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.504+0000 7f3aaf780700 1 -- 172.21.15.22:0/787535248 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3aa80619c0 con 0x7f3aa80ff9e0 2024-03-24T09:48:25.505 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.505+0000 7f3aa67fc700 1 -- 172.21.15.22:0/787535248 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f3a9805bb60 con 0x7f3aa80ff9e0 2024-03-24T09:48:25.505 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T09:48:25.505 INFO:teuthology.orchestra.run.smithi022.stdout: "mon": { 2024-03-24T09:48:25.506 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 18.2.2-952-g738c2fd6 (738c2fd67dbc0b17e2e99568a03974712d998229) reef (stable)": 2 2024-03-24T09:48:25.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T09:48:25.506 INFO:teuthology.orchestra.run.smithi022.stdout: "mgr": { 2024-03-24T09:48:25.506 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 18.2.2-952-g738c2fd6 (738c2fd67dbc0b17e2e99568a03974712d998229) reef (stable)": 2 2024-03-24T09:48:25.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T09:48:25.506 INFO:teuthology.orchestra.run.smithi022.stdout: "osd": { 2024-03-24T09:48:25.506 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 18.2.2-952-g738c2fd6 (738c2fd67dbc0b17e2e99568a03974712d998229) reef (stable)": 6 2024-03-24T09:48:25.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T09:48:25.506 INFO:teuthology.orchestra.run.smithi022.stdout: "mds": { 2024-03-24T09:48:25.506 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 18.2.2-952-g738c2fd6 (738c2fd67dbc0b17e2e99568a03974712d998229) reef (stable)": 4 2024-03-24T09:48:25.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2024-03-24T09:48:25.507 INFO:teuthology.orchestra.run.smithi022.stdout: "overall": { 2024-03-24T09:48:25.507 INFO:teuthology.orchestra.run.smithi022.stdout: "ceph version 18.2.2-952-g738c2fd6 (738c2fd67dbc0b17e2e99568a03974712d998229) reef (stable)": 14 2024-03-24T09:48:25.507 INFO:teuthology.orchestra.run.smithi022.stdout: } 2024-03-24T09:48:25.507 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T09:48:25.507 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.507+0000 7f3aaf780700 1 -- 172.21.15.22:0/787535248 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f3a9006cce0 msgr2=0x7f3a9006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:48:25.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.507+0000 7f3aaf780700 1 --2- 172.21.15.22:0/787535248 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f3a9006cce0 0x7f3a9006f1a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c009710 tx=0x7f3a9c006c60 comp rx=0 tx=0).stop 2024-03-24T09:48:25.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.507+0000 7f3aaf780700 1 -- 172.21.15.22:0/787535248 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3aa80ff9e0 msgr2=0x7f3aa8069af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:48:25.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.507+0000 7f3aaf780700 1 --2- 172.21.15.22:0/787535248 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3aa80ff9e0 0x7f3aa8069af0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f3a98006130 tx=0x7f3a98003ce0 comp rx=0 tx=0).stop 2024-03-24T09:48:25.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.507+0000 7f3aaf780700 1 -- 172.21.15.22:0/787535248 shutdown_connections 2024-03-24T09:48:25.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.507+0000 7f3aaf780700 1 --2- 172.21.15.22:0/787535248 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f3a9006cce0 0x7f3a9006f1a0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:48:25.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.507+0000 7f3aaf780700 1 --2- 172.21.15.22:0/787535248 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3aa80ff9e0 0x7f3aa8069af0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:48:25.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.507+0000 7f3aaf780700 1 --2- 172.21.15.22:0/787535248 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3aa8100be0 0x7f3aa819b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:48:25.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.507+0000 7f3aaf780700 1 -- 172.21.15.22:0/787535248 >> 172.21.15.22:0/787535248 conn(0x7f3aa80faf80 msgr2=0x7f3aa8104230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:48:25.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.507+0000 7f3aaf780700 1 -- 172.21.15.22:0/787535248 shutdown_connections 2024-03-24T09:48:25.508 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.507+0000 7f3aaf780700 1 -- 172.21.15.22:0/787535248 wait complete. 2024-03-24T09:48:25.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.612+0000 7f0bf2dee700 1 -- 172.21.15.22:0/1733291063 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0bec101560 msgr2=0x7f0bec103950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:48:25.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.612+0000 7f0bf2dee700 1 --2- 172.21.15.22:0/1733291063 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0bec101560 0x7f0bec103950 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f0be0009910 tx=0x7f0be0009c20 comp rx=0 tx=0).stop 2024-03-24T09:48:25.614 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.612+0000 7f0bf2dee700 1 -- 172.21.15.22:0/1733291063 shutdown_connections 2024-03-24T09:48:25.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.612+0000 7f0bf2dee700 1 --2- 172.21.15.22:0/1733291063 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bec103e90 0x7f0bec106280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:48:25.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.612+0000 7f0bf2dee700 1 --2- 172.21.15.22:0/1733291063 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0bec101560 0x7f0bec103950 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:48:25.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.612+0000 7f0bf2dee700 1 -- 172.21.15.22:0/1733291063 >> 172.21.15.22:0/1733291063 conn(0x7f0bec0faf70 msgr2=0x7f0bec0fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:48:25.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.612+0000 7f0bf2dee700 1 -- 172.21.15.22:0/1733291063 shutdown_connections 2024-03-24T09:48:25.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.613+0000 7f0bf2dee700 1 -- 172.21.15.22:0/1733291063 wait complete. 2024-03-24T09:48:25.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.613+0000 7f0bf2dee700 1 Processor -- start 2024-03-24T09:48:25.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.613+0000 7f0bf2dee700 1 -- start start 2024-03-24T09:48:25.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.613+0000 7f0bf2dee700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bec101560 0x7f0bec100dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:48:25.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.613+0000 7f0bf2dee700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0bec103e90 0x7f0bec0ff510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:48:25.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.613+0000 7f0bf2dee700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bec1013f0 con 0x7f0bec103e90 2024-03-24T09:48:25.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.613+0000 7f0bf2dee700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bec0ffa50 con 0x7f0bec101560 2024-03-24T09:48:25.615 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.613+0000 7f0bebfff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0bec103e90 0x7f0bec0ff510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:48:25.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.613+0000 7f0bebfff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0bec103e90 0x7f0bec0ff510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42388/0 (socket says 172.21.15.22:42388) 2024-03-24T09:48:25.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.613+0000 7f0bebfff700 1 -- 172.21.15.22:0/3922699685 learned_addr learned my addr 172.21.15.22:0/3922699685 (peer_addr_for_me v2:172.21.15.22:0/0) 2024-03-24T09:48:25.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.614+0000 7f0bebfff700 1 -- 172.21.15.22:0/3922699685 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bec101560 msgr2=0x7f0bec100dd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-24T09:48:25.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.614+0000 7f0bebfff700 1 --2- 172.21.15.22:0/3922699685 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bec101560 0x7f0bec100dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:48:25.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.614+0000 7f0bebfff700 1 -- 172.21.15.22:0/3922699685 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be00095f0 con 0x7f0bec103e90 2024-03-24T09:48:25.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.614+0000 7f0bebfff700 1 --2- 172.21.15.22:0/3922699685 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0bec103e90 0x7f0bec0ff510 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f0bd800b4d0 tx=0x7f0bd800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:48:25.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.614+0000 7f0be9ffb700 1 -- 172.21.15.22:0/3922699685 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0bd8011800 con 0x7f0bec103e90 2024-03-24T09:48:25.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.614+0000 7f0be9ffb700 1 -- 172.21.15.22:0/3922699685 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0bd8011e20 con 0x7f0bec103e90 2024-03-24T09:48:25.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.614+0000 7f0bf2dee700 1 -- 172.21.15.22:0/3922699685 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bec0ffd00 con 0x7f0bec103e90 2024-03-24T09:48:25.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.614+0000 7f0be9ffb700 1 -- 172.21.15.22:0/3922699685 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0bd800f2e0 con 0x7f0bec103e90 2024-03-24T09:48:25.616 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.614+0000 7f0bf2dee700 1 -- 172.21.15.22:0/3922699685 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bec1002d0 con 0x7f0bec103e90 2024-03-24T09:48:25.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.615+0000 7f0bf2dee700 1 -- 172.21.15.22:0/3922699685 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bec04ea80 con 0x7f0bec103e90 2024-03-24T09:48:25.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.615+0000 7f0be9ffb700 1 -- 172.21.15.22:0/3922699685 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 19) v1 ==== 90842+0+0 (secure 0 0 0) 0x7f0bd8010380 con 0x7f0bec103e90 2024-03-24T09:48:25.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.616+0000 7f0be9ffb700 1 --2- 172.21.15.22:0/3922699685 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f0bdc06cce0 0x7f0bdc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:48:25.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.616+0000 7f0be9ffb700 1 -- 172.21.15.22:0/3922699685 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f0bd808bd80 con 0x7f0bec103e90 2024-03-24T09:48:25.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.616+0000 7f0bf0b8a700 1 --2- 172.21.15.22:0/3922699685 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f0bdc06cce0 0x7f0bdc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:48:25.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.616+0000 7f0bf0b8a700 1 --2- 172.21.15.22:0/3922699685 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f0bdc06cce0 0x7f0bdc06f1a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0be0000c00 tx=0x7f0be0005fb0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:48:25.620 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.621+0000 7f0be9ffb700 1 -- 172.21.15.22:0/3922699685 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0bd8059c00 con 0x7f0bec103e90 2024-03-24T09:48:25.846 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.846+0000 7f0bf2dee700 1 -- 172.21.15.22:0/3922699685 --> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0bec19e7b0 con 0x7f0bdc06cce0 2024-03-24T09:48:25.847 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.847+0000 7f0be9ffb700 1 -- 172.21.15.22:0/3922699685 <== mgr.14215 v2:172.21.15.22:6800/773863266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f0bec19e7b0 con 0x7f0bdc06cce0 2024-03-24T09:48:25.847 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2024-03-24T09:48:25.847 INFO:teuthology.orchestra.run.smithi022.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919", 2024-03-24T09:48:25.847 INFO:teuthology.orchestra.run.smithi022.stdout: "in_progress": true, 2024-03-24T09:48:25.847 INFO:teuthology.orchestra.run.smithi022.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-24T09:48:25.848 INFO:teuthology.orchestra.run.smithi022.stdout: "services_complete": [], 2024-03-24T09:48:25.848 INFO:teuthology.orchestra.run.smithi022.stdout: "progress": "0/2 daemons upgraded", 2024-03-24T09:48:25.848 INFO:teuthology.orchestra.run.smithi022.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 image on host smithi155", 2024-03-24T09:48:25.848 INFO:teuthology.orchestra.run.smithi022.stdout: "is_paused": false 2024-03-24T09:48:25.848 INFO:teuthology.orchestra.run.smithi022.stdout:} 2024-03-24T09:48:25.849 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.849+0000 7f0bf2dee700 1 -- 172.21.15.22:0/3922699685 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f0bdc06cce0 msgr2=0x7f0bdc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:48:25.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.849+0000 7f0bf2dee700 1 --2- 172.21.15.22:0/3922699685 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f0bdc06cce0 0x7f0bdc06f1a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0be0000c00 tx=0x7f0be0005fb0 comp rx=0 tx=0).stop 2024-03-24T09:48:25.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.849+0000 7f0bf2dee700 1 -- 172.21.15.22:0/3922699685 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0bec103e90 msgr2=0x7f0bec0ff510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:48:25.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.849+0000 7f0bf2dee700 1 --2- 172.21.15.22:0/3922699685 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0bec103e90 0x7f0bec0ff510 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f0bd800b4d0 tx=0x7f0bd800b890 comp rx=0 tx=0).stop 2024-03-24T09:48:25.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.849+0000 7f0bf2dee700 1 -- 172.21.15.22:0/3922699685 shutdown_connections 2024-03-24T09:48:25.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.849+0000 7f0bf2dee700 1 --2- 172.21.15.22:0/3922699685 >> [v2:172.21.15.22:6800/773863266,v1:172.21.15.22:6801/773863266] conn(0x7f0bdc06cce0 0x7f0bdc06f1a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:48:25.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.849+0000 7f0bf2dee700 1 --2- 172.21.15.22:0/3922699685 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bec101560 0x7f0bec100dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:48:25.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.849+0000 7f0bf2dee700 1 --2- 172.21.15.22:0/3922699685 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0bec103e90 0x7f0bec0ff510 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:48:25.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.849+0000 7f0bf2dee700 1 -- 172.21.15.22:0/3922699685 >> 172.21.15.22:0/3922699685 conn(0x7f0bec0faf70 msgr2=0x7f0bec0fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:48:25.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.849+0000 7f0bf2dee700 1 -- 172.21.15.22:0/3922699685 shutdown_connections 2024-03-24T09:48:25.850 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:48:25.849+0000 7f0bf2dee700 1 -- 172.21.15.22:0/3922699685 wait complete. 2024-03-24T09:48:44.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:44.371+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:44.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:44.371+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:44.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:44.371+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:45.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:45.010+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:45.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:45.010+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:45.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:45.010+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:45.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:45.376+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:45.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:45.376+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:45.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:45.376+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:46.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:46.019+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:46.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:46.019+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:46.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:46.019+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:46.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:45.903+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:46.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:45.903+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:46.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:45.903+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:46.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:46.372+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:46.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:46.372+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:46.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:46.372+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:47.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:47.053+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:47.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:47.053+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:47.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:47.053+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:47.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:46.942+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:47.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:46.942+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:47.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:46.942+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:47.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:47.390+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:47.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:47.390+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:47.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:47.390+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:48.090 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:47.932+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:48.090 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:47.932+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:48.090 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:47.932+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:48.399 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:48.090+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:48.400 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:48.090+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:48.400 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:48.090+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:48.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:48.399+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:48.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:48.399+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:48.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:48.399+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:49.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:48.982+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:49.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:48.982+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:49.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:48.982+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:49.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:49.043+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:49.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:49.043+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:49.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:49.043+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:49.659 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:49.399+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:49.659 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:49.399+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:49.660 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:49.399+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:50.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:49.996+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:50.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:49.996+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:50.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:49.996+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:50.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:49.969+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:50.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:49.969+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:50.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:49.969+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:50.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:50.382+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:50.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:50.382+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:50.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:50.382+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:51.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:50.980+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:51.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:50.980+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:51.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:50.980+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:51.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:50.996+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:51.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:50.996+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:51.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:50.996+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:51.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:51.386+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:51.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:51.386+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:51.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:51.386+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:52.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:52.014+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:52.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:52.014+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:52.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:52.014+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:52.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:51.986+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:52.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:51.986+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:52.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:51.986+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:52.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:52.339+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:52.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:52.339+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:52.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:52.339+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:53.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:52.975+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:53.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:52.975+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:53.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:52.975+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:53.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:53.012+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:53.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:53.012+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:53.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:53.012+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:53.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:53.294+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:53.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:53.294+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:53.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:53.294+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:54.249 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:53.990+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:54.249 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:53.990+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:54.249 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:53.990+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:54.249 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:54.039+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:54.249 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:54.039+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:54.250 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:54.039+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:54.531 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:54.249+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:54.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:54.249+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:54.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:54.249+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:55.261 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:55.052+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:55.262 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:55.052+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:55.262 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:55.052+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:55.262 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:55.008+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:55.262 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:55.008+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:55.262 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:55.008+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:55.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:55.261+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:55.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:55.261+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:55.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:55.261+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:56.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:56.223+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:56.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:56.223+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:56.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:56.223+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:56.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:56.038+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:56.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:56.038+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:56.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:56.038+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:56.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:56.049+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:56.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:56.049+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:56.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:56.049+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:57.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:57.195+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:57.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:57.195+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:57.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:57.195+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:57.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:56.991+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:57.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:56.991+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:57.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:56.991+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:57.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:57.015+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:57.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:57.015+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:57.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:57.015+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:58.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:58.216+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:58.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:58.216+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:58.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:58.217+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:58.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:58.039+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:58.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:58.039+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:58.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:58.039+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:58.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:57.980+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:58.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:57.980+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:58.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:57.980+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:59.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:59.194+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:59.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:59.194+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:59.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:48:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:48:59.194+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:48:59.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:59.083+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:59.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:59.083+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:59.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:48:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:48:59.083+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:48:59.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:58.993+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:59.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:58.993+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:48:59.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:48:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:48:58.993+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:00.182 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:00.027+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:00.183 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:00.027+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:00.183 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:00.027+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:00.183 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:00.101+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:00.183 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:00.101+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:00.183 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:00.101+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:00.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:00.203+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:00.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:00.203+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:00.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:00.203+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:01.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:01.222+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:01.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:01.222+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:01.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:01.222+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:01.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:01.142+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:01.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:01.142+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:01.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:01.142+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:01.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:01.022+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:01.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:01.022+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:01.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:01.022+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:02.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:02.203+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:02.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:02.203+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:02.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:02.203+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:02.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:02.171+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:02.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:02.171+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:02.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:02.171+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:02.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:02.014+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:02.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:02.014+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:02.285 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:02.014+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:03.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:03.204+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:03.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:03.204+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:03.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:03.204+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:03.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:03.208+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:03.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:03.208+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:03.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:03.208+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:03.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:03.009+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:03.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:03.009+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:03.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:03.009+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:04.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:04.190+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:04.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:04.190+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:04.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:04.190+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:04.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:04.173+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:04.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:04.173+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:04.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:04.173+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:04.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:04.018+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:04.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:04.018+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:04.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:04.018+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:05.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:05.202+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:05.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:05.203+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:05.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:05.203+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:05.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:05.061+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:05.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:05.061+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:05.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:05.061+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:05.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:05.179+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:05.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:05.179+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:05.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:05.179+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:06.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:06.174+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:06.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:06.174+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:06.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:06.174+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:06.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:06.198+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:06.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:06.198+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:06.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:06.198+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:06.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:06.079+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:06.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:06.079+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:06.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:06.079+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:07.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:07.142+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:07.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:07.142+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:07.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:07.142+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:07.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:07.075+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:07.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:07.076+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:07.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:07.076+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:07.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:07.158+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:07.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:07.158+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:07.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:07.158+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:08.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:08.162+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:08.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:08.162+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:08.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:08.162+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:08.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:08.122+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:08.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:08.122+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:08.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:08.122+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:08.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:08.041+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:08.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:08.041+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:08.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:08.041+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:09.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:09.116+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:09.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:09.116+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:09.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:09.116+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:09.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:09.120+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:09.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:09.120+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:09.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:09.120+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:09.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:09.081+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:09.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:09.081+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:09.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:09.081+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:10.174 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:10.096+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:10.174 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:10.096+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:10.175 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:10.096+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:10.175 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:10.088+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:10.175 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:10.088+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:10.175 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:10.088+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:10.176 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:10.064+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:10.176 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:10.064+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:10.176 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:10.064+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:11.318 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:11.051+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:11.318 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:11.051+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:11.318 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:11.051+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:11.319 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:11.080+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:11.319 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:11.080+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:11.319 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:11.080+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:11.319 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:11.070+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:11.319 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:11.070+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:11.320 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:11.070+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:12.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:12.035+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:12.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:12.035+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:12.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:12.035+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:12.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:12.010+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:12.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:12.010+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:12.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:12.010+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:12.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:12.104+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:12.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:12.104+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:12.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:12.104+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:13.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:13.053+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:13.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:13.053+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:13.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:13.053+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:13.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:13.102+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:13.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:13.102+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:13.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:13.102+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:13.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:13.012+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:13.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:13.012+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:13.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:13.012+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:14.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:14.024+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:14.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:14.024+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:14.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:14.024+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:14.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:14.139+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:14.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:14.139+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:14.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:14.139+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:14.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:13.974+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:14.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:13.974+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:14.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:13.974+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:15.194 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:14.981+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:15.194 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:14.981+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:15.194 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:14.981+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:15.194 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:15.159+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:15.194 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:15.159+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:15.194 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:15.159+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:15.195 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:14.935+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:15.195 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:14.935+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:15.195 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:14.935+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:16.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:16.029+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:16.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:16.029+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:16.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:16.029+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:16.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:16.117+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:16.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:16.117+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:16.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:16.117+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:16.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:15.943+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:16.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:15.943+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:16.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:15.943+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:17.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:16.992+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:17.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:16.992+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:17.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:16.992+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:17.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:17.109+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:17.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:17.109+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:17.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:17.109+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:17.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:16.985+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:17.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:16.985+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:17.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:16.985+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:18.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:17.986+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:18.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:17.986+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:18.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:17.986+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:18.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:18.113+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:18.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:18.113+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:18.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:18.113+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:18.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:18.008+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:18.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:18.009+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:18.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:18.009+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:19.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:18.953+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:19.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:18.953+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:19.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:18.953+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:19.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:19.158+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:19.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:19.158+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:19.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:19.158+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:19.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:19.034+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:19.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:19.034+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:19.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:19.034+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:20.162 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:20.028+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:20.162 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:20.028+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:20.163 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:20.028+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:20.163 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:19.945+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:20.163 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:19.945+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:20.163 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:19.945+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:20.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:20.162+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:20.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:20.162+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:20.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:20.162+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:21.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:20.987+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:21.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:20.987+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:21.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:20.987+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:21.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:21.180+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:21.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:21.180+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:21.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:21.180+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:21.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:20.979+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:21.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:20.979+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:21.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:20.979+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:22.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:22.172+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:22.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:22.172+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:22.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:22.172+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:22.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:21.999+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:22.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:21.999+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:22.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:21.999+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:22.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:22.018+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:22.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:22.018+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:22.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:22.018+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:23.512 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:23.041+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:23.512 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:23.041+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:23.512 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:23.041+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:23.513 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:23.068+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:23.513 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:23.068+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:23.513 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:23.068+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:23.513 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:23.186+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:23.513 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:23.186+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:23.514 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:23.186+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:24.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:24.141+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:24.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:24.141+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:24.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:24.141+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:24.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:24.116+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:24.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:24.116+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:24.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:24.116+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:24.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:24.000+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:24.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:24.000+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:24.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:24.000+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:25.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:25.129+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:25.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:25.129+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:25.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:25.129+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:25.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:24.977+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:25.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:24.977+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:25.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:24.977+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:25.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:25.106+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:25.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:25.106+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:25.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:25.106+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:26.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:26.147+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:26.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:26.147+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:26.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:26.147+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:26.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:26.096+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:26.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:26.096+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:26.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:26.096+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:26.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:25.939+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:26.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:25.939+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:26.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:25.939+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:27.176 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:27.058+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:27.176 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:27.058+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:27.176 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:27.058+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:27.177 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:26.923+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:27.177 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:26.923+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:27.177 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:26.923+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:27.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:27.175+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:27.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:27.175+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:27.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:27.175+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:28.182 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:28.045+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:28.182 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:28.045+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:28.182 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:28.045+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:28.182 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:27.915+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:28.182 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:27.915+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:28.182 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:27.915+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:28.181+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:28.181+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:28.181+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:29.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:29.161+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:29.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:29.161+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:29.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:29.161+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:29.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:29.045+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:29.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:29.045+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:29.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:29.045+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:29.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:28.931+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:29.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:28.931+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:29.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:28.931+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:30.127 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:30.039+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:30.127 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:30.039+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:30.127 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:30.039+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:30.128 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:29.902+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:30.128 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:29.902+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:30.128 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:29.902+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:30.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:30.127+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:30.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:30.127+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:30.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:30.127+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:31.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:31.079+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:31.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:31.079+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:31.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:31.079+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:31.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:31.019+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:31.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:31.019+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:31.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:31.019+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:31.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:30.891+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:31.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:30.891+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:31.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:30.891+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:32.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:32.040+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:32.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:32.040+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:32.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:32.040+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:32.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:32.067+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:32.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:32.067+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:32.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:32.067+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:32.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:31.885+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:32.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:31.885+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:32.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:31.885+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:33.201 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:33.028+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:33.201 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:33.028+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:33.201 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:33.028+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:33.202 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:32.888+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:33.202 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:32.888+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:33.202 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:32.888+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:33.202 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:33.039+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:33.203 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:33.039+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:33.203 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:33.039+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:34.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:34.041+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:34.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:34.041+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:34.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:34.041+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:34.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:34.018+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:34.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:34.018+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:34.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:34.018+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:34.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:33.854+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:34.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:33.854+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:34.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:33.854+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:35.186 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:35.083+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:35.186 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:35.083+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:35.186 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:35.083+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:35.186 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:34.891+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:35.187 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:34.891+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:35.187 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:34.891+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:35.187 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:34.989+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:35.187 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:34.989+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:35.187 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:34.989+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:36.272 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:36.035+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:36.272 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:36.035+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:36.273 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:36.035+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:36.273 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:36.115+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:36.273 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:36.115+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:36.273 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:36.115+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:36.273 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:35.914+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:36.273 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:35.914+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:36.274 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:35.914+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:37.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:37.049+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:37.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:37.049+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:37.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:37.049+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:37.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:37.124+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:37.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:37.124+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:37.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:37.124+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:37.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:36.936+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:37.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:36.936+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:37.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:36.936+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:38.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:38.062+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:38.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:38.062+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:38.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:38.062+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:38.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:37.906+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:38.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:37.906+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:38.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:37.906+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:38.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:38.137+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:38.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:38.137+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:38.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:38.137+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:39.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:39.079+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:39.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:39.079+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:39.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:39.079+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:39.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:39.145+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:39.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:39.145+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:39.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:39.145+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:39.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:38.941+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:39.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:38.941+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:39.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:38.941+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:40.140 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:40.051+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:40.140 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:40.051+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:40.140 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:40.051+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:40.140 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:39.920+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:40.140 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:39.920+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:40.141 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:39.920+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:40.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:40.139+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:40.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:40.139+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:40.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:40.139+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:41.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:41.144+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:41.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:41.144+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:41.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:41.144+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:41.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:40.925+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:41.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:40.925+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:41.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:40.925+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:41.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:41.083+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:41.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:41.083+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:41.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:41.083+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:42.129 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:41.876+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:42.129 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:41.876+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:42.129 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:41.876+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:42.417 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:42.179+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:42.417 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:42.179+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:42.417 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:42.179+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:42.417 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:42.128+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:42.417 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:42.128+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:42.418 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:42.128+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:43.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:43.130+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:43.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:43.130+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:43.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:43.130+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:43.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:43.126+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:43.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:43.126+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:43.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:43.126+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:43.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:42.900+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:43.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:42.900+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:43.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:42.900+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:44.169 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:44.133+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:44.169 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:44.133+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:44.169 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:44.133+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:44.170 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:43.915+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:44.170 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:43.915+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:44.170 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:43.915+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:44.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:44.169+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:44.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:44.169+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:44.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:44.169+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:45.142 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:44.890+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:45.142 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:44.890+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:45.142 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:44.890+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:45.488 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:45.178+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:45.488 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:45.178+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:45.488 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:45.178+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:45.488 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:45.143+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:45.489 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:45.143+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:45.489 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:45.143+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:46.216 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:45.896+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:46.217 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:45.896+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:46.217 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:45.896+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:46.217 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:46.143+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:46.217 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:46.143+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:46.217 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:46.143+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:46.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:46.216+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:46.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:46.216+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:46.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:46.216+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:47.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:46.892+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:47.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:46.892+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:47.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:46.892+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:47.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:47.224+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:47.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:47.224+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:47.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:47.224+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:47.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:47.158+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:47.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:47.158+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:47.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:47.158+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:48.233 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:48.150+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:48.233 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:48.150+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:48.234 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:48.150+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:48.234 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:47.912+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:48.234 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:47.912+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:48.234 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:47.912+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:48.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:48.233+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:48.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:48.233+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:48.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:48.233+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:49.211 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:49.115+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:49.211 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:49.115+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:49.212 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:49.115+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:49.212 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:48.884+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:49.212 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:48.884+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:49.212 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:48.884+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:49.212 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:49.211+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:49.213 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:49.211+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:49.213 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:49.211+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:50.138 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:49.882+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:50.139 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:49.882+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:50.139 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:49.882+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:50.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:50.139+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:50.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:50.139+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:50.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:50.139+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:50.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:50.252+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:50.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:50.252+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:50.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:50.252+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:51.147 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:50.884+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:51.147 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:50.884+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:51.147 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:50.884+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:51.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:51.261+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:51.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:51.261+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:51.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:51.261+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:51.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:51.147+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:51.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:51.147+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:51.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:51.147+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:52.187 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:51.907+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:52.188 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:51.907+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:52.188 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:51.907+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:52.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:52.262+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:52.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:52.262+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:52.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:52.262+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:52.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:52.188+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:52.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:52.188+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:52.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:52.188+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:53.220 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:52.881+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:53.221 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:52.881+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:53.221 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:52.881+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:53.514 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:53.236+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:53.514 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:53.236+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:53.514 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:53.236+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:53.514 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:53.221+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:53.514 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:53.221+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:53.514 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:53.221+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:54.178 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:53.874+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:54.178 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:53.874+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:54.178 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:53.874+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:54.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:54.261+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:54.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:54.261+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:54.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:54.261+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:54.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:54.179+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:54.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:54.179+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:54.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:54.179+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:55.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:54.832+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:55.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:54.832+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:55.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:54.832+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:55.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:55.225+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:55.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:55.225+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:55.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:55.225+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:55.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:55.173+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:55.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:55.173+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:55.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:55.173+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:56.153 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:55.804+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:56.154 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:55.804+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:56.154 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:55.804+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:56.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:56.256+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:56.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:56.256+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:56.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:56.256+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:56.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:56.154+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:56.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:56.154+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:56.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:56.154+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:57.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:56.758+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:57.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:56.758+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:57.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:56.758+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:57.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:57.301+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:57.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:57.301+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:57.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:57.301+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:57.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:57.172+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:57.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:57.172+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:57.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:57.172+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:58.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:57.776+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:58.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:57.776+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:58.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:57.776+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:58.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:58.139+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:58.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:58.139+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:58.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:58.139+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:58.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:58.342+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:58.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:58.342+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:58.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:58.342+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:59.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:58.766+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:59.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:58.766+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:59.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:58.766+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:49:59.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:59.385+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:59.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:59.385+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:59.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:49:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:49:59.385+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:49:59.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:59.143+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:59.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:59.143+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:49:59.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:49:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:49:59.143+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:00.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:59.767+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:00.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:59.768+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:00.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:49:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:49:59.768+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:00.422 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:00.147+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:00.422 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:00.147+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:00.422 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:00.147+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:00.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:00.421+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:00.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:00.421+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:00.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:00.421+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:01.103 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:00.790+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:01.103 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:00.790+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:01.103 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:00.790+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:01.417 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:01.103+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:01.417 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:01.103+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:01.417 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:01.103+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:01.756 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:01.417+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:01.757 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:01.417+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:01.757 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:01.417+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:02.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:01.756+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:02.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:01.756+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:02.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:01.756+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:02.430 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:02.077+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:02.431 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:02.077+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:02.431 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:02.077+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:02.752 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:02.430+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:02.752 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:02.430+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:02.752 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:02.430+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:03.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:02.751+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:03.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:02.751+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:03.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:02.751+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:03.443 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:03.101+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:03.443 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:03.101+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:03.443 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:03.101+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:03.729 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:03.443+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:03.729 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:03.443+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:03.729 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:03.443+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:04.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:03.729+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:04.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:03.729+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:04.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:03.729+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:04.429 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:04.069+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:04.429 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:04.069+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:04.429 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:04.069+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:04.730 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:04.429+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:04.730 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:04.429+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:04.730 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:04.429+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:05.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:04.729+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:05.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:04.729+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:05.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:04.729+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:05.473 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:05.058+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:05.474 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:05.058+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:05.474 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:05.058+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:05.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:05.473+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:05.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:05.473+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:05.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:05.473+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:05.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:05.711+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:05.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:05.711+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:05.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:05.711+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:06.505 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:06.044+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:06.505 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:06.045+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:06.505 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:06.045+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:06.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:06.678+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:06.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:06.678+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:06.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:06.678+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:06.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:06.504+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:06.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:06.504+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:06.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:06.504+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:07.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:07.065+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:07.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:07.065+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:07.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:07.065+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:07.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:07.489+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:07.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:07.489+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:07.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:07.489+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:07.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:07.717+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:07.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:07.717+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:07.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:07.717+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:08.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:08.016+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:08.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:08.016+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:08.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:08.016+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:08.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:08.500+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:08.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:08.500+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:08.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:08.500+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:08.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:08.719+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:08.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:08.719+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:08.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:08.719+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:09.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:08.984+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:09.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:08.984+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:09.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:08.984+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:09.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:09.709+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:09.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:09.709+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:09.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:09.709+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:09.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:09.530+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:09.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:09.530+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:09.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:09.530+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:10.187 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:10.022+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:10.188 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:10.022+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:10.188 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:10.022+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:11.023 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:10.561+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:11.023 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:10.561+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:11.023 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:10.561+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:11.023 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:10.728+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:11.023 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:10.728+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:11.024 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:10.728+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:11.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:11.022+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:11.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:11.022+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:11.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:11.022+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:11.899 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:11.611+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:11.899 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:11.611+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:11.900 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:11.611+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:11.900 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:11.695+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:11.900 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:11.695+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:11.900 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:11.695+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:12.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:12.072+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:12.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:12.072+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:12.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:12.072+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:13.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:12.568+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:13.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:12.568+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:13.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:12.568+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:13.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:12.660+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:13.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:12.660+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:13.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:12.660+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:13.515 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:13.113+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:13.515 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:13.113+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:13.515 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:13.113+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:13.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:13.596+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:13.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:13.596+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:13.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:13.596+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:13.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:13.662+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:13.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:13.662+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:13.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:13.662+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:14.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:14.137+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:14.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:14.137+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:14.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:14.137+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:15.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:14.644+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:15.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:14.644+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:15.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:14.644+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:15.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:14.707+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:15.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:14.707+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:15.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:14.707+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:15.299 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:15.135+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:15.299 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:15.135+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:15.299 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:15.135+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:16.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:15.627+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:16.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:15.627+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:16.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:15.627+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:16.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:15.711+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:16.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:15.711+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:16.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:15.711+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:16.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:16.098+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:16.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:16.098+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:16.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:16.098+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:16.947 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:16.648+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:16.948 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:16.648+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:16.948 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:16.648+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:16.948 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:16.662+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:16.948 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:16.662+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:16.948 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:16.662+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:17.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:17.108+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:17.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:17.108+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:17.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:17.108+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:18.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:17.601+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:18.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:17.601+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:18.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:17.601+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:18.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:17.666+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:18.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:17.666+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:18.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:17.666+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:18.531 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:18.137+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:18.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:18.137+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:18.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:18.137+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:19.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:18.703+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:19.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:18.703+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:19.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:18.703+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:19.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:18.581+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:19.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:18.581+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:19.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:18.581+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:19.531 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:19.087+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:19.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:19.087+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:19.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:19.087+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:19.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:19.592+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:19.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:19.592+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:19.855 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:19.592+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:19.855 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:19.727+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:19.855 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:19.727+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:19.855 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:19.727+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:20.188 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:20.074+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:20.188 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:20.074+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:20.188 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:20.074+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:21.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:20.577+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:21.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:20.577+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:21.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:20.577+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:21.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:20.755+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:21.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:20.755+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:21.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:20.755+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:21.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:21.098+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:21.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:21.098+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:21.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:21.098+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:21.994 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:21.575+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:21.995 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:21.575+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:21.995 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:21.575+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:21.995 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:21.713+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:21.995 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:21.713+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:21.995 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:21.713+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:22.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:22.066+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:22.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:22.066+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:22.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:22.066+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:23.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:22.619+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:23.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:22.619+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:23.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:22.619+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:23.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:22.752+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:23.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:22.752+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:23.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:22.752+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:23.515 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:23.078+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:23.515 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:23.078+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:23.515 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:23.078+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:23.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:23.627+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:23.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:23.627+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:23.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:23.627+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:23.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:23.738+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:23.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:23.738+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:23.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:23.738+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:24.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:24.031+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:24.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:24.031+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:24.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:24.031+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:24.988 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:24.677+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:24.989 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:24.677+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:24.989 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:24.677+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:24.989 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:24.725+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:24.989 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:24.725+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:24.989 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:24.725+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:25.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:50:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:50:25.179+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 1 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:50:25.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:24.988+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:25.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:24.988+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:25.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:24.988+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:25.987 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:25.705+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:25.987 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:25.705+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:25.987 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:25.705+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:25.988 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:25.677+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:25.988 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:25.677+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:25.988 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:25.677+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:26.281 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:25.987+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:26.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:25.987+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:26.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:25.987+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:26.950 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:26.660+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:26.951 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:26.660+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:26.951 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:26.660+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:26.951 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:26.659+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:26.951 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:26.659+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:26.951 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:26.659+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:27.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:26.950+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:27.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:26.950+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:27.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:26.950+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:27.913 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:27.652+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:27.914 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:27.652+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:27.914 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:27.652+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:27.914 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:27.610+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:27.914 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:27.610+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:27.914 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:27.610+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:28.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:27.913+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:28.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:27.913+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:28.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:27.913+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:28.899 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:28.635+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:28.899 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:28.635+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:28.899 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:28.635+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:28.899 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:28.648+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:28.900 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:28.648+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:28.900 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:28.648+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:29.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:28.898+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:29.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:28.898+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:29.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:28.898+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:30.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:29.686+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:30.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:29.686+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:30.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:29.686+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:30.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:29.876+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:30.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:29.876+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:30.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:29.876+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:30.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:29.648+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:30.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:29.648+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:30.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:29.649+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:30.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:50:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:50:30.179+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 11 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:50:30.917 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:30.697+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:30.917 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:30.697+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:30.917 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:30.697+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:30.917 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:30.655+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:30.917 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:30.655+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:30.917 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:30.655+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:31.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:30.916+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:31.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:30.916+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:31.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:30.916+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:32.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:31.669+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:32.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:31.669+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:32.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:31.669+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:32.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:31.868+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:32.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:31.868+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:32.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:31.868+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:32.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:31.661+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:32.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:31.661+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:32.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:31.661+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:33.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:32.859+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:33.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:32.859+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:33.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:32.859+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:33.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:32.696+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:33.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:32.696+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:33.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:32.696+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:33.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:32.689+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:33.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:32.689+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:33.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:32.689+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:33.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:33.698+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:33.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:33.698+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:33.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:33.698+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:33.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:33.729+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:33.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:33.729+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:33.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:33.729+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:34.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:33.889+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:34.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:33.889+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:34.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:33.889+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:35.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:34.708+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:35.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:34.708+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:35.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:34.708+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:35.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:34.779+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:35.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:34.779+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:35.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:34.779+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:35.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:34.840+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:35.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:34.840+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:35.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:34.840+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:35.301 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:50:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:50:35.179+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 22 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:50:36.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:35.755+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:36.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:35.755+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:36.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:35.755+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:36.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:35.879+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:36.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:35.879+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:36.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:35.879+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:36.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:35.739+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:36.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:35.739+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:36.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:35.739+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:37.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:36.768+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:37.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:36.768+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:37.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:36.768+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:37.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:36.832+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:37.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:36.832+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:37.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:36.832+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:37.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:36.732+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:37.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:36.732+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:37.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:36.732+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:38.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:37.810+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:38.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:37.810+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:38.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:37.810+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:38.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:37.815+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:38.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:37.815+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:38.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:37.815+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:38.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:37.757+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:38.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:37.757+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:38.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:37.757+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:39.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:38.808+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:39.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:38.808+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:39.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:38.808+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:38.816+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:39.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:38.816+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:39.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:38.816+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:39.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:38.763+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:39.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:38.763+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:39.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:38.763+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:40.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:39.798+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:40.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:39.798+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:40.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:39.798+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:40.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:39.789+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:40.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:39.789+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:40.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:39.789+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:40.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:39.756+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:40.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:39.756+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:40.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:39.756+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:40.531 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:50:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:50:40.180+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 32 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:50:41.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:40.825+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:41.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:40.825+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:41.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:40.825+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:41.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:40.743+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:41.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:40.743+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:41.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:40.743+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:41.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:40.722+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:41.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:40.722+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:41.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:40.722+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:42.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:41.818+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:42.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:41.818+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:42.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:41.818+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:42.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:41.750+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:42.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:41.750+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:42.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:41.750+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:42.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:41.749+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:42.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:41.749+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:42.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:41.749+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:43.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:42.853+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:43.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:42.853+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:43.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:42.853+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:43.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:42.727+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:43.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:42.727+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:43.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:42.727+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:43.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:42.751+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:43.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:42.751+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:43.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:42.751+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:43.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:43.733+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:43.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:43.733+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:43.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:43.733+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:43.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:43.744+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:43.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:43.744+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:43.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:43.744+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:44.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:43.894+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:44.314 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:43.894+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:44.314 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:43.894+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:44.931 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:44.690+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:44.931 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:44.690+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:44.931 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:44.690+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:44.932 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:44.706+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:44.932 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:44.706+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:44.932 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:44.706+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:45.188 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:50:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:50:45.180+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 58 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:50:45.188 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:44.931+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:45.189 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:44.931+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:45.189 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:44.931+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:46.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:45.919+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:46.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:45.919+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:46.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:45.919+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:46.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:45.710+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:46.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:45.710+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:46.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:45.710+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:46.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:45.705+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:46.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:45.705+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:46.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:45.705+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:46.930 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:46.676+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:46.930 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:46.676+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:46.930 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:46.676+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:46.931 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:46.734+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:46.931 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:46.734+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:46.931 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:46.734+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:47.243 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:46.943+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:47.244 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:46.943+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:47.244 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:46.943+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:47.936 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:47.627+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:47.937 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:47.627+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:47.937 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:47.627+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:47.937 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:47.778+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:47.937 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:47.778+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:47.937 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:47.778+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:48.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:47.936+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:48.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:47.936+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:48.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:47.936+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:48.972 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:48.824+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:48.972 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:48.824+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:48.972 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:48.824+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:48.972 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:48.605+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:48.972 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:48.605+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:48.972 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:48.605+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:49.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:48.971+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:49.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:48.971+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:49.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:48.971+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:49.855 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:49.803+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:49.855 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:49.803+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:49.855 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:49.803+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:49.855 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:49.584+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:49.855 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:49.584+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:49.856 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:49.584+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:50.181 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:49.978+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:50.182 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:49.978+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:50.182 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:49.978+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:50.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:50:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:50:50.181+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 86 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:50:51.015 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:50.597+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:51.015 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:50.597+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:51.015 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:50.597+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:51.015 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:50.834+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:51.016 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:50.834+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:51.016 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:50.834+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:51.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:51.014+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:51.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:51.014+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:51.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:51.014+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:52.003 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:51.824+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:52.004 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:51.824+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:52.004 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:51.824+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:52.004 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:51.620+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:52.004 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:51.620+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:52.004 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:51.620+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:52.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:52.003+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:52.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:52.003+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:52.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:52.003+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:53.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:52.622+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:53.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:52.622+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:53.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:52.622+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:53.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:52.866+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:53.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:52.866+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:53.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:52.866+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:53.517 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:53.043+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:53.517 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:53.043+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:53.518 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:53.043+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:53.781 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:53.634+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:53.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:53.634+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:53.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:53.634+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:54.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:53.857+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:54.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:53.857+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:54.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:53.857+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:54.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:54.047+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:54.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:54.047+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:54.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:54.047+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:54.893 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:54.631+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:54.893 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:54.631+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:54.894 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:54.631+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:55.182 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:55.006+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:55.183 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:55.006+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:55.183 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:55.006+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:55.183 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:54.893+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:55.183 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:54.893+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:55.183 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:54.893+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:55.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:50:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:50:55.181+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 115 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:50:56.028 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:55.872+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:56.028 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:55.872+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:56.029 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:55.872+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:56.029 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:55.671+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:56.029 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:55.671+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:56.029 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:55.671+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:56.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:56.028+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:56.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:56.028+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:56.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:56.028+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:56.984 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:56.676+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:56.984 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:56.677+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:56.984 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:56.677+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:56.985 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:56.829+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:56.985 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:56.829+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:56.985 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:56.829+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:57.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:56.983+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:57.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:56.983+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:57.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:56.983+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:57.977 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:57.709+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:57.977 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:57.709+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:57.977 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:57.709+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:57.977 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:57.821+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:57.977 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:57.821+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:57.977 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:57.821+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:58.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:57.976+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:58.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:57.976+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:58.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:57.976+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:58.975 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:58.713+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:58.975 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:58.713+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:58.975 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:58.713+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:50:58.975 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:58.782+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:58.975 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:58.782+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:58.975 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:58.782+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:50:59.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:58.974+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:59.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:58.974+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:50:59.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:50:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:50:58.974+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:00.024 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:59.777+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:00.025 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:59.777+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:00.025 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:50:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:50:59.777+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:00.025 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:59.680+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:00.025 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:59.680+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:00.025 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:50:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:50:59.680+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:00.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:51:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:51:00.182+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 145 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:51:00.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:00.024+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:00.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:00.024+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:00.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:00.024+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:01.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:00.730+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:01.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:00.730+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:01.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:00.730+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:01.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:00.692+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:01.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:00.692+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:01.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:00.692+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:01.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:01.061+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:01.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:01.061+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:01.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:01.061+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:02.015 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:01.714+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:02.015 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:01.714+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:02.016 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:01.714+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:02.016 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:01.753+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:02.016 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:01.753+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:02.016 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:01.753+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:02.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:02.015+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:02.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:02.015+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:02.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:02.015+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:02.968 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:02.699+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:02.968 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:02.699+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:02.968 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:02.699+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:02.968 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:02.771+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:02.968 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:02.771+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:02.968 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:02.771+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:03.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:02.968+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:03.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:02.968+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:03.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:02.968+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:03.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:03.682+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:03.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:03.682+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:03.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:03.682+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:03.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:03.766+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:03.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:03.766+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:03.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:03.766+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:04.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:03.979+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:04.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:03.979+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:04.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:03.979+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:04.969 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:04.754+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:04.969 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:04.754+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:04.969 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:04.754+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:04.969 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:04.653+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:04.969 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:04.653+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:04.970 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:04.653+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:05.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:51:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:51:05.182+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 175 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:51:05.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:04.969+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:05.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:04.969+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:05.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:04.969+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:05.967 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:05.754+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:05.967 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:05.754+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:05.967 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:05.754+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:05.968 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:05.677+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:05.968 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:05.677+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:05.968 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:05.677+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:06.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:05.967+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:06.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:05.967+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:06.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:05.967+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:06.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:06.686+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:06.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:06.686+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:06.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:06.686+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:06.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:06.722+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:06.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:06.722+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:06.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:06.722+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:07.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:06.982+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:07.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:06.982+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:07.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:06.982+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:07.692 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:07.651+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:07.692 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:07.651+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:07.692 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:07.651+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:07.981 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:07.693+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:07.982 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:07.693+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:07.982 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:07.693+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:08.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:07.982+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:08.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:07.982+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:08.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:07.982+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:08.973 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:08.674+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:08.973 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:08.674+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:08.973 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:08.674+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:08.973 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:08.610+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:08.973 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:08.610+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:08.974 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:08.610+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:09.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:08.973+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:09.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:08.973+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:09.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:08.973+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:09.946 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:09.669+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:09.946 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:09.669+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:09.946 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:09.669+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:09.947 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:09.608+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:09.947 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:09.608+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:09.947 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:09.608+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:10.203 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:51:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:51:10.184+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 202 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:51:10.204 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:09.947+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:10.204 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:09.947+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:10.204 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:09.947+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:10.946 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:10.712+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:10.947 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:10.712+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:10.947 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:10.712+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:10.947 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:10.645+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:10.947 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:10.645+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:10.947 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:10.645+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:11.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:10.947+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:11.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:10.947+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:11.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:10.947+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:11.918 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:11.629+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:11.918 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:11.629+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:11.918 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:11.629+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:11.918 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:11.699+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:11.919 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:11.699+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:11.919 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:11.699+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:12.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:11.918+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:12.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:11.918+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:12.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:11.918+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:12.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:12.661+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:12.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:12.661+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:12.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:12.661+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:12.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:12.652+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:12.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:12.652+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:12.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:12.652+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:13.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:12.933+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:13.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:12.933+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:13.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:12.933+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:13.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:13.656+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:13.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:13.656+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:13.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:13.656+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:13.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:13.662+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:13.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:13.662+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:13.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:13.662+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:14.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:13.964+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:14.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:13.964+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:14.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:13.964+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:14.998 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:14.620+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:15.017 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:14.620+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:15.017 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:14.620+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:15.018 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:14.667+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:15.018 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:14.667+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:15.018 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:14.667+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:15.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:51:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:51:15.184+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 231 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:51:15.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:14.998+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:15.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:14.998+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:15.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:14.998+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:15.976 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:15.589+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:15.976 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:15.589+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:15.976 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:15.589+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:15.976 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:15.666+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:15.977 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:15.666+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:15.977 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:15.666+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:16.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:16.036+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:16.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:16.037+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:16.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:16.037+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:17.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:16.593+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:17.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:16.593+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:17.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:16.593+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:17.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:16.692+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:17.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:16.692+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:17.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:16.692+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:17.527 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:17.085+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:17.527 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:17.085+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:17.527 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:17.085+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:17.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:17.702+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:17.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:17.702+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:17.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:17.702+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:17.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:17.552+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:17.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:17.552+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:17.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:17.552+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:18.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:18.068+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:18.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:18.068+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:18.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:18.068+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:19.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:18.561+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:19.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:18.561+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:19.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:18.561+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:19.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:18.683+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:19.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:18.683+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:19.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:18.683+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:19.516 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:19.103+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:19.516 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:19.103+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:19.517 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:19.103+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:19.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:19.517+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:19.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:19.517+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:19.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:19.517+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:19.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:19.647+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:19.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:19.647+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:19.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:19.647+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:20.492 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:51:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:51:20.184+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 257 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:51:20.492 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:20.123+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:20.492 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:20.123+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:20.493 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:20.123+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:20.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:20.492+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:20.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:20.492+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:20.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:20.492+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:20.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:20.628+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:20.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:20.628+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:20.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:20.628+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:21.444 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:21.161+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:21.445 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:21.161+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:21.445 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:21.161+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:21.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:21.650+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:21.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:21.650+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:21.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:21.650+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:21.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:21.444+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:21.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:21.444+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:21.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:21.444+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:22.436 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:22.113+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:22.437 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:22.113+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:22.437 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:22.113+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:22.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:22.678+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:22.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:22.678+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:22.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:22.678+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:22.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:22.436+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:22.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:22.436+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:22.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:22.436+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:23.459 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:23.118+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:23.459 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:23.118+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:23.459 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:23.118+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:23.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:23.458+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:23.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:23.458+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:23.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:23.458+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:23.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:23.670+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:23.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:23.670+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:23.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:23.670+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:24.497 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:24.088+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:24.497 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:24.088+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:24.498 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:24.088+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:24.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:24.497+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:24.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:24.497+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:24.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:24.497+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:24.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:24.701+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:24.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:24.701+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:24.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:24.701+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:25.486 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:51:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:51:25.184+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 287 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:51:25.486 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:25.116+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:25.486 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:25.116+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:25.486 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:25.116+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:25.740 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:25.485+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:25.740 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:25.486+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:25.740 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:25.486+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:26.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:25.739+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:26.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:25.739+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:26.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:25.739+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:26.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:26.127+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:26.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:26.127+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:26.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:26.127+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:27.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:26.534+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:27.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:26.534+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:27.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:26.534+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:27.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:26.734+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:27.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:26.734+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:27.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:26.734+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:27.486 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:27.166+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:27.487 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:27.166+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:27.487 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:27.166+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:27.778 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:27.486+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:27.778 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:27.486+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:27.778 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:27.486+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:28.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:27.778+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:28.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:27.778+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:28.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:27.778+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:28.522 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:28.124+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:28.522 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:28.124+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:28.522 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:28.124+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:28.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:28.522+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:28.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:28.522+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:28.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:28.522+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:29.154 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:28.817+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:29.155 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:28.817+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:29.155 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:28.817+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:29.511 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:29.154+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:29.511 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:29.154+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:29.511 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:29.154+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:29.773 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:29.511+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:29.773 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:29.511+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:29.774 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:29.511+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:30.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:29.773+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:30.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:29.773+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:30.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:29.773+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:30.464 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:51:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:51:30.185+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 315 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:51:30.464 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:30.128+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:30.464 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:30.128+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:30.464 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:30.128+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:30.735 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:30.463+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:30.735 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:30.463+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:30.735 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:30.463+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:31.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:30.734+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:31.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:30.734+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:31.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:30.734+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:31.439 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:31.145+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:31.439 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:31.145+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:31.439 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:31.145+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:31.766 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:31.438+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:31.766 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:31.438+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:31.766 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:31.438+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:32.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:31.765+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:32.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:31.765+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:32.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:31.765+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:32.476 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:32.159+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:32.476 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:32.159+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:32.476 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:32.159+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:32.762 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:32.476+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:32.762 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:32.476+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:32.762 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:32.476+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:33.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:32.761+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:33.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:32.761+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:33.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:32.761+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:33.519 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:33.174+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:33.520 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:33.174+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:33.520 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:33.174+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:33.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:33.726+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:33.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:33.726+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:33.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:33.726+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:33.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:33.519+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:33.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:33.519+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:33.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:33.519+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:34.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:34.201+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:34.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:34.201+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:34.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:34.201+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:35.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:34.738+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:35.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:34.738+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:35.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:34.738+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:35.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:34.738+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.24303.0:5799 3.8 3:122e4596:::100000044bd.00000000:head [write 0~2176 in=2176b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:35.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:34.547+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:35.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:34.547+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:35.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:34.547+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:35.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:51:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:51:35.185+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 343 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:51:35.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:35.248+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:35.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:35.248+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:35.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:35.248+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:35.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:35.249+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:36.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:35.743+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:36.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:35.743+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:36.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:35.743+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:36.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:35.744+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:36.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:35.554+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:36.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:35.554+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:36.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:35.554+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:36.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:35.556+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:36.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:36.244+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:36.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:36.244+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:36.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:36.244+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:36.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:36.245+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:37.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:36.744+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:37.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:36.744+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:37.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:36.744+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:37.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:36.745+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:37.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:36.600+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:37.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:36.600+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:37.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:36.600+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:37.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:36.601+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:37.557 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:37.290+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:37.557 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:37.290+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:37.557 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:37.290+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:37.558 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:37.291+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:38.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:37.701+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:38.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:37.701+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:38.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:37.701+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:38.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:37.701+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:38.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:37.557+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:38.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:37.557+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:38.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:37.557+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:38.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:37.557+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:38.709 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:38.546+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:38.709 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:38.546+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:38.709 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:38.546+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:38.709 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:38.546+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:38.709 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:38.318+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:38.709 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:38.318+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:38.709 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:38.318+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:38.709 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:38.319+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:39.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:38.708+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:39.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:38.708+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:39.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:38.708+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:39.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:38.709+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:39.756 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:39.514+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:39.756 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:39.514+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:39.756 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:39.514+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:39.756 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:39.515+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:39.756 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:39.354+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:39.757 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:39.354+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:39.757 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:39.354+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:39.757 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:39.355+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:40.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:39.755+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:40.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:39.755+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:40.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:39.755+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:40.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:39.757+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:40.486 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:51:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:51:40.186+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 370 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:51:40.487 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:40.335+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:40.487 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:40.335+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:40.487 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:40.335+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:40.487 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:40.336+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:40.780 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:40.485+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:40.780 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:40.485+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:40.780 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:40.485+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:40.781 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:40.487+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:41.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:40.779+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:41.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:40.779+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:41.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:40.779+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:41.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:40.780+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:41.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:41.509+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:41.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:41.509+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:41.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:41.509+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:41.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:41.510+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:41.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:41.319+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:41.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:41.319+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:41.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:41.319+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:41.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:41.320+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:42.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:41.800+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:42.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:41.800+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:42.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:41.800+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:42.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:41.801+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:42.770 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:42.342+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:42.771 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:42.342+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:42.771 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:42.342+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:42.771 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:42.343+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:42.771 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:42.531+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:42.771 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:42.531+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:42.772 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:42.531+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:42.772 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:42.532+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:43.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:42.823+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:43.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:42.823+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:43.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:42.823+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:43.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:42.824+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:43.780 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:43.326+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:43.780 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:43.326+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:43.780 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:43.326+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:43.780 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:43.326+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:43.780 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:43.493+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:43.780 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:43.493+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:43.780 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:43.493+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:43.780 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:43.493+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:44.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:43.779+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:44.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:43.779+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:44.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:43.779+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:44.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:43.780+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:44.759 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:44.337+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:44.759 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:44.337+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:44.760 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:44.337+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:44.760 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:44.338+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:44.760 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:44.517+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:44.760 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:44.517+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:44.760 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:44.517+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:44.760 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:44.518+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:45.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:44.759+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:45.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:44.759+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:45.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:44.759+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:45.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:44.760+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:45.469 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:51:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:51:45.186+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 404 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:51:45.469 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:45.323+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:45.469 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:45.323+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:45.469 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:45.323+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:45.469 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:45.324+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:45.738 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:45.469+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:45.750 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:45.469+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:45.750 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:45.469+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:45.750 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:45.469+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:46.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:45.738+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:46.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:45.738+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:46.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:45.738+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:46.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:45.739+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:46.768 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:46.286+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:46.769 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:46.286+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:46.769 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:46.286+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:46.769 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:46.287+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:46.769 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:46.468+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:46.769 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:46.468+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:46.769 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:46.468+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:46.769 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:46.468+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:47.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:46.768+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:47.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:46.768+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:47.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:46.768+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:47.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:46.769+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:47.725 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:47.296+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:47.725 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:47.296+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:47.725 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:47.296+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:47.725 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:47.297+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:47.725 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:47.450+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:47.725 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:47.450+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:47.725 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:47.450+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:47.726 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:47.451+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:48.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:47.724+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:48.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:47.724+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:48.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:47.724+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:48.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:47.725+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:48.759 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:48.337+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:48.759 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:48.337+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:48.759 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:48.337+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:48.759 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:48.338+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:48.759 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:48.478+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:48.759 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:48.478+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:48.759 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:48.478+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:48.759 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:48.478+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:49.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:48.758+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:49.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:48.758+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:49.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:48.758+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:49.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:48.759+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:49.744 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:49.306+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:49.744 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:49.306+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:49.744 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:49.306+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:49.745 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:49.306+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:49.745 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:49.502+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:49.745 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:49.502+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:49.745 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:49.502+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:49.745 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:49.503+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:50.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:49.744+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:50.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:49.744+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:50.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:49.744+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:50.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:49.745+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:50.458 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:51:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:51:50.187+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 437 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:51:50.459 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:50.341+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:50.459 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:50.341+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:50.459 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:50.341+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:50.459 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:50.342+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:50.717 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:50.458+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:50.717 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:50.458+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:50.717 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:50.458+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:50.717 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:50.459+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:51.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:50.716+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:51.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:50.716+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:51.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:50.716+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:51.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:50.717+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:51.633 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:51.340+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:51.633 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:51.340+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:51.633 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:51.340+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:51.633 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:51.341+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:51.633 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:51.474+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:51.634 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:51.474+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:51.634 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:51.474+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:51.634 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:51.475+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:52.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:51.698+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:52.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:51.698+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:52.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:51.698+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:52.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:51.699+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:52.708 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:52.323+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:52.709 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:52.323+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:52.709 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:52.323+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:52.709 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:52.324+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:52.709 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:52.500+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:52.709 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:52.500+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:52.709 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:52.500+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:52.709 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:52.501+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:53.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:52.708+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:53.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:52.708+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:53.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:52.708+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:53.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:52.709+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:53.665 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:53.331+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:53.665 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:53.331+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:53.666 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:53.331+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:53.666 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:53.332+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:53.666 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:53.454+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:53.666 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:53.454+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:53.666 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:53.454+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:53.666 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:53.455+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:54.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:53.665+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:54.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:53.665+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:54.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:53.665+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:54.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:53.666+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:54.628 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:54.294+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:54.629 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:54.294+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:54.629 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:54.294+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:54.629 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:54.296+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:54.629 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:54.494+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:54.629 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:54.494+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:54.629 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:54.494+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:54.629 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:54.495+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:55.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:54.628+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:55.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:54.628+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:55.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:54.628+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:55.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:54.629+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:55.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:51:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:51:55.187+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 467 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:51:55.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:55.311+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:55.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:55.311+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:55.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:55.311+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:55.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:55.311+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:56.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:55.534+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:56.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:55.534+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:56.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:55.534+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:56.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:55.535+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:56.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:55.628+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:56.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:55.628+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:56.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:55.628+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:56.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:55.629+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:56.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:56.341+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:56.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:56.341+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:56.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:56.341+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:56.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:56.342+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:56.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:56.580+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:56.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:56.580+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:56.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:56.580+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:56.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:56.581+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:56.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:56.585+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:56.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:56.585+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:56.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:56.585+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:56.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:56.586+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:57.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:57.386+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:57.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:57.386+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:57.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:57.386+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:57.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:57.387+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:57.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:57.574+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:57.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:57.574+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:57.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:57.574+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:57.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:57.574+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:57.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:57.594+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:57.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:57.594+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:57.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:57.594+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:57.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:57.595+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:58.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:58.407+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:58.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:58.407+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:58.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:58.407+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:58.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:58.408+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:58.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:58.529+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:58.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:58.529+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:58.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:58.529+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:58.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:58.530+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:58.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:58.577+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:58.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:58.577+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:58.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:58.577+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:58.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:58.577+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:59.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:59.390+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:59.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:59.390+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:59.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:59.390+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:51:59.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:51:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:51:59.391+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:59.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:59.549+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:59.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:59.549+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:59.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:59.549+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:51:59.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:51:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:51:59.550+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:51:59.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:59.612+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:59.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:59.612+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:59.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:59.612+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:51:59.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:51:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:51:59.613+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:00.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:52:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:52:00.188+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 496 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:52:00.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:00.372+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:00.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:00.372+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:00.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:00.372+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:00.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:00.372+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:00.849 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:00.643+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:00.849 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:00.643+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:00.849 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:00.643+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:00.849 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:00.644+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:00.849 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:00.568+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:00.849 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:00.568+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:00.849 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:00.568+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:00.849 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:00.569+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:01.694 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:01.407+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:01.694 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:01.407+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:01.694 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:01.407+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:01.694 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:01.408+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:01.694 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:01.520+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:01.694 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:01.520+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:01.694 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:01.520+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:01.695 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:01.521+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:02.031 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:01.693+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:02.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:01.693+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:02.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:01.693+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:02.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:01.694+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:02.650 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:02.537+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:02.650 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:02.537+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:02.650 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:02.537+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:02.651 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:02.538+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:02.651 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:02.360+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:02.651 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:02.360+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:02.651 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:02.360+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:02.651 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:02.360+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:02.961 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:02.650+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:02.961 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:02.650+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:02.961 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:02.650+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:02.961 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:02.651+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:03.678 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:03.335+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:03.678 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:03.335+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:03.678 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:03.335+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:03.678 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:03.335+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:03.678 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:03.514+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:03.679 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:03.514+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:03.679 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:03.514+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:03.679 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:03.514+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:04.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:03.677+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:04.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:03.677+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:04.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:03.677+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:04.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:03.678+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:04.647 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:04.557+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:04.647 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:04.557+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:04.648 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:04.557+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:04.648 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:04.557+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:04.648 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:04.379+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:04.648 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:04.379+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:04.648 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:04.379+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:04.648 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:04.380+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:05.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:04.647+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:05.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:04.647+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:05.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:04.647+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:05.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:04.647+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:05.514 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:52:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:52:05.188+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 525 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:52:05.514 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:05.387+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:05.514 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:05.387+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:05.514 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:05.387+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:05.514 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:05.388+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:05.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:05.670+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:05.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:05.670+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:05.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:05.670+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:05.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:05.671+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:05.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:05.513+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:05.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:05.513+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:05.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:05.513+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:05.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:05.514+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:06.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:06.391+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:06.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:06.391+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:06.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:06.391+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:06.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:06.392+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:06.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:06.503+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:06.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:06.503+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:06.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:06.503+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:06.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:06.504+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:06.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:06.627+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:06.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:06.627+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:06.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:06.627+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:06.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:06.628+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:07.674 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:07.377+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:07.674 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:07.377+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:07.674 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:07.377+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:07.674 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:07.377+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:07.675 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:07.529+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:07.675 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:07.529+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:07.675 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:07.529+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:07.675 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:07.530+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:08.011 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:07.673+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:08.011 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:07.673+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:08.011 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:07.673+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:08.011 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:07.675+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:08.642 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:08.370+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:08.642 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:08.370+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:08.642 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:08.370+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:08.642 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:08.370+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:08.643 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:08.542+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:08.643 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:08.542+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:08.643 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:08.542+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:08.643 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:08.543+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:09.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:08.641+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:09.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:08.641+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:09.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:08.641+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:09.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:08.642+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:09.637 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:09.362+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:09.638 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:09.362+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:09.638 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:09.362+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:09.638 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:09.363+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:09.638 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:09.590+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:09.638 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:09.590+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:09.638 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:09.590+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:09.638 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:09.592+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:10.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:09.637+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:10.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:09.637+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:10.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:09.637+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:10.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:09.638+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:10.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:52:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:52:10.189+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 553 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:52:10.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:10.363+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:10.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:10.363+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:10.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:10.363+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:10.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:10.363+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:11.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:10.549+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:11.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:10.549+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:11.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:10.549+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:11.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:10.549+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:11.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:10.661+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:11.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:10.661+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:11.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:10.661+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:11.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:10.662+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:11.628 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:11.335+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:11.628 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:11.335+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:11.628 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:11.335+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:11.628 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:11.336+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:11.629 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:11.541+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:11.629 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:11.541+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:11.629 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:11.541+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:11.629 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:11.543+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:12.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:11.628+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:12.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:11.628+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:12.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:11.628+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:12.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:11.629+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:12.568 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:12.296+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:12.568 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:12.296+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:12.568 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:12.296+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:12.568 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:12.297+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:13.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:12.674+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:13.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:12.674+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:13.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:12.674+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:13.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:12.675+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:13.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:12.567+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:13.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:12.567+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:13.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:12.567+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:13.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:12.569+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:13.628 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:13.335+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:13.628 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:13.335+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:13.628 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:13.335+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:13.628 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:13.337+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:13.628 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:13.566+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:13.628 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:13.566+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:13.629 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:13.566+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:13.629 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:13.567+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:14.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:13.628+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:14.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:13.628+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:14.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:13.628+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:14.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:13.629+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:14.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:14.380+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:14.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:14.380+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:14.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:14.380+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:14.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:14.381+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:14.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:14.608+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:14.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:14.608+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:14.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:14.608+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:14.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:14.609+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:14.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:14.577+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:14.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:14.577+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:14.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:14.577+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:14.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:14.578+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:15.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:52:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:52:15.189+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 581 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:52:15.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:15.365+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:15.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:15.365+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:15.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:15.365+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:15.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:15.366+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:16.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:15.592+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:16.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:15.592+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:16.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:15.592+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:16.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:15.593+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:16.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:15.536+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:16.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:15.536+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:16.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:15.536+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:16.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:15.537+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:16.508 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:16.381+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:16.508 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:16.381+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:16.508 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:16.381+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:16.508 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:16.382+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:16.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:16.643+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:16.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:16.643+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:16.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:16.643+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:16.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:16.643+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:16.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:16.507+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:16.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:16.507+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:16.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:16.507+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:16.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:16.508+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:17.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:17.542+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:17.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:17.542+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:17.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:17.542+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:17.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:17.542+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:17.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:17.383+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:17.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:17.383+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:17.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:17.383+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:17.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:17.385+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:17.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:17.611+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:17.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:17.611+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:17.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:17.611+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:17.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:17.612+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:18.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:18.355+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:18.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:18.355+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:18.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:18.355+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:18.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:18.356+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:19.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:18.598+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:19.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:18.598+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:19.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:18.598+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:19.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:18.599+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:19.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:18.585+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:19.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:18.585+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:19.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:18.585+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:19.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:18.586+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:19.578 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:19.307+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:19.578 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:19.307+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:19.578 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:19.307+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:19.578 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:19.308+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:19.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:19.577+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:19.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:19.577+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:19.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:19.577+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:19.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:19.578+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:19.855 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:19.586+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:19.855 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:19.586+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:19.855 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:19.586+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:19.855 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:19.587+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:20.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:52:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:52:20.189+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 608 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:52:20.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:20.272+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:20.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:20.272+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:20.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:20.272+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:20.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:20.272+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:21.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:20.622+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:21.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:20.622+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:21.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:20.622+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:21.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:20.622+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:21.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:20.564+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:21.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:20.564+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:21.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:20.564+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:21.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:20.565+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:21.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:21.278+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:21.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:21.278+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:21.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:21.278+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:21.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:21.278+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:22.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:21.552+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:22.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:21.552+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:22.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:21.552+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:22.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:21.553+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:22.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:21.621+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:22.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:21.621+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:22.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:21.621+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:22.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:21.622+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:22.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:22.257+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:22.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:22.257+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:22.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:22.257+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:22.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:22.258+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:23.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:22.647+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:23.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:22.647+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:23.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:22.647+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:23.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:22.648+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:23.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:22.602+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:23.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:22.602+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:23.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:22.602+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:23.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:22.603+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:23.523 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:23.209+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:23.524 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:23.209+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:23.524 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:23.209+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:23.524 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:23.210+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:23.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:23.677+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:23.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:23.677+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:23.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:23.677+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:23.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:23.678+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:23.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:23.559+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:23.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:23.560+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:23.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:23.560+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:23.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:23.560+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:24.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:24.210+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:24.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:24.210+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:24.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:24.210+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:24.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:24.211+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:25.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:24.538+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:25.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:24.538+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:25.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:24.538+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:25.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:24.539+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:25.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:24.648+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:25.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:24.648+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:25.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:24.648+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:25.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:24.649+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:25.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:52:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:52:25.190+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 637 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:52:25.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:25.229+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:25.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:25.229+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:25.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:25.229+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:25.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:25.229+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:26.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:25.544+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:26.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:25.544+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:26.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:25.544+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:26.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:25.544+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:26.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:25.639+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:26.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:25.639+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:26.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:25.639+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:26.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:25.639+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:26.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:26.240+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:26.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:26.240+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:26.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:26.240+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:26.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:26.241+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:27.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:26.635+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:27.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:26.635+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:27.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:26.635+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:27.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:26.635+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:27.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:26.592+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:27.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:26.592+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:27.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:26.592+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:27.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:26.593+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:27.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:27.276+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:27.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:27.276+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:27.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:27.276+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:27.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:27.276+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:28.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:27.673+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:28.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:27.673+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:28.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:27.673+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:28.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:27.674+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:28.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:27.550+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:28.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:27.550+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:28.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:27.550+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:28.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:27.551+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:28.517 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:28.319+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:28.517 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:28.319+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:28.517 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:28.319+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:28.517 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:28.320+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:28.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:28.644+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:28.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:28.644+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:28.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:28.644+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:28.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:28.645+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:28.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:28.517+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:28.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:28.517+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:28.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:28.517+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:28.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:28.518+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:29.664 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:29.506+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:29.664 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:29.506+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:29.664 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:29.506+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:29.664 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:29.507+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:29.664 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:29.362+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:29.664 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:29.362+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:29.665 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:29.362+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:29.665 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:29.363+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:30.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:29.664+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:30.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:29.664+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:30.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:29.664+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:30.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:29.665+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:30.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:52:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:52:30.192+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 665 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:52:30.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:30.396+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:30.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:30.396+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:30.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:30.396+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:30.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:30.397+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:31.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:30.536+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:31.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:30.536+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:31.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:30.536+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:31.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:30.537+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:31.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:30.643+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:31.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:30.644+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:31.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:30.644+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:31.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:30.644+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:31.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:31.516+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:31.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:31.516+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:31.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:31.516+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:31.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:31.517+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:31.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:31.446+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:31.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:31.446+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:31.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:31.446+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:31.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:31.447+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:31.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:31.694+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:31.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:31.694+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:31.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:31.694+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:31.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:31.695+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:32.709 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:32.419+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:32.710 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:32.419+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:32.710 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:32.419+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:32.710 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:32.421+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:32.710 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:32.502+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:32.710 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:32.502+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:32.710 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:32.502+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:32.710 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:32.503+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:33.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:32.710+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:33.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:32.710+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:33.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:32.710+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:33.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:32.711+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:33.524 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:33.379+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:33.524 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:33.379+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:33.524 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:33.379+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:33.524 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:33.379+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:33.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:33.703+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:33.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:33.703+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:33.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:33.703+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:33.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:33.704+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:33.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:33.537+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:33.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:33.537+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:33.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:33.537+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:33.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:33.538+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:34.640 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:34.375+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:34.640 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:34.375+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:34.640 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:34.375+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:34.640 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:34.375+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:34.641 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:34.551+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:34.641 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:34.551+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:34.641 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:34.551+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:34.641 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:34.552+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:35.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:34.656+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:35.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:34.656+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:35.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:34.656+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:35.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:34.657+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:35.315 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:52:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:52:35.192+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 697 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:52:35.660 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:35.414+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:35.660 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:35.414+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:35.660 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:35.414+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:35.660 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:35.415+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:35.660 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:35.532+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:35.660 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:35.532+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:35.660 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:35.532+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:35.660 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:35.533+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:36.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:35.660+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:36.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:35.660+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:36.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:35.660+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:36.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:35.661+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:36.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:36.580+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:36.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:36.580+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:36.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:36.580+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:36.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:36.581+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:36.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:36.404+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:36.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:36.405+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:36.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:36.405+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:36.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:36.405+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:36.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:36.642+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:36.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:36.642+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:36.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:36.642+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:36.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:36.643+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:37.713 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:37.443+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:37.713 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:37.443+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:37.713 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:37.443+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:37.713 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:37.444+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:37.714 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:37.692+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:37.714 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:37.692+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:37.714 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:37.692+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:37.714 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:37.693+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:37.714 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:37.626+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:37.714 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:37.626+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:37.714 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:37.626+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:37.714 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:37.627+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:38.658 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:38.434+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:38.658 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:38.434+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:38.658 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:38.434+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:38.658 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:38.435+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:39.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:38.659+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:39.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:38.659+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:39.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:38.659+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:39.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:38.659+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:38.690+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:38.690+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:38.690+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:39.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:38.691+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:39.662 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:39.396+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:39.662 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:39.396+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:39.662 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:39.396+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:39.662 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:39.397+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:40.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:39.662+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:40.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:39.662+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:40.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:39.662+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:40.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:39.663+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:40.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:39.682+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:40.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:39.682+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:40.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:39.682+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:40.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:39.683+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:40.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:52:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:52:40.193+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 730 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:52:40.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:40.440+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:40.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:40.440+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:40.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:40.440+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:40.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:40.441+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:41.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:40.662+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:41.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:40.662+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:41.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:40.662+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:41.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:40.663+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:41.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:40.707+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:41.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:40.707+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:41.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:40.707+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:41.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:40.707+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:41.710 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:41.394+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:41.710 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:41.394+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:41.710 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:41.394+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:41.710 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:41.395+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:42.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:41.716+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:42.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:41.716+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:42.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:41.716+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:42.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:41.717+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:42.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:41.709+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:42.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:41.709+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:42.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:41.709+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:42.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:41.710+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:42.748 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:42.382+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:42.748 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:42.382+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:42.748 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:42.382+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:42.749 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:42.383+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:43.010 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:42.748+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:43.010 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:42.748+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:43.010 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:42.748+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:43.010 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:42.748+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:43.010 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:42.763+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:43.011 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:42.763+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:43.011 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:42.763+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:43.011 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:42.764+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:43.722 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:43.388+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:43.722 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:43.388+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:43.722 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:43.388+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:43.722 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:43.388+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:44.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:43.721+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:44.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:43.721+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:44.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:43.721+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:44.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:43.722+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:44.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:43.801+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:44.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:43.801+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:44.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:43.801+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:44.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:43.802+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:44.692 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:44.377+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:44.692 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:44.377+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:44.693 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:44.377+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:44.693 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:44.378+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:45.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:44.692+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:45.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:44.692+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:45.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:44.692+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:45.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:44.693+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:45.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:44.783+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:45.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:44.783+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:45.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:44.783+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:45.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:44.783+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:45.314 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:52:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:52:45.192+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 759 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:52:45.690 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:45.365+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:45.690 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:45.365+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:45.691 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:45.365+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:45.691 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:45.366+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:46.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:45.690+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:46.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:45.690+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:46.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:45.690+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:46.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:45.691+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:46.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:45.779+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:46.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:45.779+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:46.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:45.779+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:46.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:45.780+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:46.672 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:46.326+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:46.672 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:46.326+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:46.672 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:46.326+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:46.672 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:46.326+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:46.929 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:46.671+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:46.929 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:46.671+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:46.929 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:46.671+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:46.929 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:46.672+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:46.930 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:46.824+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:46.930 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:46.824+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:46.930 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:46.824+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:46.930 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:46.825+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:47.629 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:47.298+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:47.630 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:47.298+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:47.630 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:47.298+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:47.630 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:47.299+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:48.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:47.629+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:48.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:47.629+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:48.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:47.629+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:48.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:47.630+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:48.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:47.788+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:48.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:47.788+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:48.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:47.788+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:48.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:47.789+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:48.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:48.268+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:48.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:48.268+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:48.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:48.268+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:48.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:48.269+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:49.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:48.593+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:49.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:48.593+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:49.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:48.593+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:49.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:48.593+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:49.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:48.782+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:49.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:48.782+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:49.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:48.782+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:49.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:48.783+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:49.531 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:49.254+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:49.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:49.254+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:49.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:49.254+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:49.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:49.255+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:49.857 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:49.584+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:49.857 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:49.584+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:49.857 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:49.584+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:49.857 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:49.585+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:49.857 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:49.738+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:49.857 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:49.738+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:49.857 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:49.738+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:49.857 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:49.738+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:50.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:52:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:52:50.193+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 787 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:52:50.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:50.243+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:50.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:50.243+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:50.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:50.243+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:50.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:50.243+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:51.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:50.632+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:51.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:50.632+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:51.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:50.632+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:51.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:50.633+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:51.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:50.693+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:51.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:50.693+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:51.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:50.693+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:51.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:50.694+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:51.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:51.212+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:51.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:51.212+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:51.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:51.212+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:51.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:51.213+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:52.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:51.585+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:52.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:51.585+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:52.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:51.585+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:52.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:51.586+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:52.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:51.652+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:52.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:51.652+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:52.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:51.652+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:52.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:51.653+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:52.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:52.177+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:52.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:52.177+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:52.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:52.177+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:52.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:52.177+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:53.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:52.542+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:53.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:52.542+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:53.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:52.542+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:53.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:52.543+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:53.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:52.666+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:53.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:52.666+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:53.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:52.666+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:53.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:52.667+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:53.441 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:53.225+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:53.441 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:53.225+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:53.441 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:53.225+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:53.441 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:53.225+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:53.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:53.585+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:53.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:53.585+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:53.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:53.585+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:53.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:53.585+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:53.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:53.631+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:53.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:53.631+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:53.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:53.631+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:53.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:53.632+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:54.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:54.249+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:54.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:54.249+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:54.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:54.249+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:54.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:54.250+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:55.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:54.603+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:55.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:54.603+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:55.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:54.603+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:55.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:54.604+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:55.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:54.598+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:55.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:54.598+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:55.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:54.598+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:55.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:54.599+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:55.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:52:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:52:55.193+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 815 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:52:55.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:55.229+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:55.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:55.229+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:55.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:55.229+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:55.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:55.230+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:56.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:55.604+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:56.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:55.604+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:56.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:55.604+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:56.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:55.605+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:56.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:55.588+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:56.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:55.588+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:56.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:55.588+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:56.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:55.588+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:56.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:56.189+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:56.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:56.189+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:56.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:56.189+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:56.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:56.190+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:57.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:56.574+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:57.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:56.574+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:57.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:56.574+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:57.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:56.575+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:57.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:56.629+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:57.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:56.629+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:57.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:56.629+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:57.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:56.629+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:57.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:57.212+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:57.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:57.212+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:57.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:57.212+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:57.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:57.213+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:58.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:57.677+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:58.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:57.677+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:58.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:57.677+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:58.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:57.678+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:58.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:57.616+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:58.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:57.616+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:58.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:57.616+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:58.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:57.617+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:58.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:58.251+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:58.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:58.251+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:58.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:58.251+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:58.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:58.252+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:59.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:58.633+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:59.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:58.633+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:59.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:58.633+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:59.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:58.634+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:59.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:58.719+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:59.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:58.719+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:59.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:58.719+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:59.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:58.720+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:59.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:59.263+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:59.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:59.263+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:59.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:59.263+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:52:59.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:52:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:52:59.264+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:59.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:59.585+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:59.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:59.585+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:59.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:59.585+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:52:59.855 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:52:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:52:59.586+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:52:59.855 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:59.729+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:59.855 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:59.729+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:59.855 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:59.729+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:52:59.855 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:52:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:52:59.730+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:00.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:53:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:53:00.193+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 842 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:53:00.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:00.216+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:00.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:00.216+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:00.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:00.216+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:00.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:00.216+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:01.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:00.568+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:01.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:00.568+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:01.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:00.568+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:01.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:00.568+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:01.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:00.765+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:01.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:00.765+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:01.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:00.765+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:01.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:00.766+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:01.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:01.185+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:01.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:01.185+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:01.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:01.185+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:01.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:01.185+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:02.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:01.596+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:02.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:01.596+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:02.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:01.596+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:02.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:01.597+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:02.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:01.740+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:02.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:01.740+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:02.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:01.740+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:02.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:01.741+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:02.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:02.192+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:02.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:02.192+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:02.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:02.192+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:02.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:02.192+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:03.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:02.785+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:03.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:02.785+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:03.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:02.785+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:03.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:02.786+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:03.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:02.604+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:03.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:02.604+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:03.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:02.604+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:03.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:02.605+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:03.526 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:03.203+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:03.526 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:03.203+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:03.527 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:03.203+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:03.527 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:03.203+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:03.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:03.637+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:03.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:03.637+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:03.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:03.637+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:03.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:03.637+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:04.168 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:03.825+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:04.168 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:03.825+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:04.168 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:03.825+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:04.168 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:03.826+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:04.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:04.167+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:04.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:04.167+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:04.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:04.167+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:04.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:04.168+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:04.855 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:04.590+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:04.855 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:04.590+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:04.855 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:04.590+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:04.855 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:04.591+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:05.182 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:04.854+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:05.183 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:04.854+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:05.183 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:04.854+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:05.183 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:04.855+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:05.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:53:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:53:05.194+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 871 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:53:05.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:05.182+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:05.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:05.182+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:05.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:05.182+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:05.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:05.183+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:05.817 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:05.624+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:05.817 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:05.624+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:05.817 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:05.624+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:05.817 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:05.625+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:06.167 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:05.817+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:06.167 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:05.817+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:06.167 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:05.817+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:06.167 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:05.818+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:06.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:06.167+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:06.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:06.167+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:06.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:06.167+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:06.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:06.168+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:07.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:06.665+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:07.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:06.665+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:07.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:06.665+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:07.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:06.665+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:07.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:06.774+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:07.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:06.774+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:07.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:06.774+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:07.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:06.775+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:07.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:07.212+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:07.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:07.212+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:07.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:07.212+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:07.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:07.213+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:08.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:07.674+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:08.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:07.674+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:08.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:07.674+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:08.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:07.675+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:08.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:07.773+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:08.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:07.773+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:08.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:07.773+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:08.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:07.774+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:08.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:08.202+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:08.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:08.202+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:08.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:08.202+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:08.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:08.203+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:09.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:08.708+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:09.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:08.708+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:09.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:08.708+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:09.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:08.709+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:09.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:08.810+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:09.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:08.810+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:09.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:08.810+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:09.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:08.811+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:09.521 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:09.174+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:09.521 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:09.174+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:09.521 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:09.174+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:09.521 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:09.175+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:10.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:09.664+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:10.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:09.664+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:10.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:09.664+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:10.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:09.665+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:10.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:09.816+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:10.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:09.816+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:10.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:09.817+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:10.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:09.817+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:10.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:53:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:53:10.194+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 899 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:53:10.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:10.158+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:10.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:10.158+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:10.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:10.158+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:10.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:10.159+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:11.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:10.714+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:11.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:10.714+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:11.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:10.714+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:11.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:10.715+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:11.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:10.830+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:11.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:10.830+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:11.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:10.830+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:11.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:10.830+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:11.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:11.170+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:11.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:11.170+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:11.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:11.170+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:11.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:11.171+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:12.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:11.692+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:12.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:11.692+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:12.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:11.692+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:12.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:11.693+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:12.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:11.878+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:12.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:11.878+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:12.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:11.878+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:12.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:11.878+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:12.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:12.216+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:12.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:12.216+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:12.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:12.216+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:12.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:12.217+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:13.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:12.725+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:13.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:12.725+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:13.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:12.725+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:13.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:12.726+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:13.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:12.893+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:13.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:12.893+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:13.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:12.893+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:13.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:12.893+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:13.527 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:13.169+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:13.527 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:13.169+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:13.527 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:13.169+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:13.527 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:13.169+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:13.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:13.772+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:13.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:13.772+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:13.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:13.772+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:13.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:13.772+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:14.145 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:13.845+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:14.145 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:13.845+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:14.146 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:13.845+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:14.146 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:13.846+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:14.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:14.145+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:14.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:14.145+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:14.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:14.145+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:14.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:14.146+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:14.864 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:14.727+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:14.864 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:14.727+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:14.864 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:14.727+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:14.864 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:14.728+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:15.181 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:14.863+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:15.181 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:14.863+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:15.181 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:14.863+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:15.181 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:14.864+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:15.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:53:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:53:15.195+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 927 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:53:15.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:15.181+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:15.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:15.181+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:15.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:15.181+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:15.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:15.182+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:16.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:15.683+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:16.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:15.683+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:16.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:15.683+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:16.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:15.683+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:16.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:15.861+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:16.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:15.861+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:16.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:15.861+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:16.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:15.862+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:16.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:16.168+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:16.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:16.168+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:16.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:16.168+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:16.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:16.169+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:17.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:16.676+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:17.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:16.676+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:17.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:16.676+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:17.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:16.677+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:17.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:16.856+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:17.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:16.857+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:17.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:16.857+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:17.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:16.858+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:17.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:17.197+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:17.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:17.197+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:17.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:17.197+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:17.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:17.199+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:18.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:17.694+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:18.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:17.694+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:18.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:17.694+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:18.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:17.695+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:18.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:17.819+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:18.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:17.819+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:18.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:17.819+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:18.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:17.819+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:18.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:18.199+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:18.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:18.199+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:18.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:18.199+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:18.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:18.200+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:19.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:18.836+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:19.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:18.836+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:19.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:18.836+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:19.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:18.836+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:19.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:18.735+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:19.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:18.735+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:19.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:18.735+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:19.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:18.735+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:19.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:19.168+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:19.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:19.168+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:19.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:19.168+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:19.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:19.169+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:20.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:19.688+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:20.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:19.688+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:20.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:19.688+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:20.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:19.689+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:20.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:19.885+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:20.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:19.885+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:20.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:19.885+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:20.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:19.886+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:20.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:53:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:53:20.195+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 954 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:53:20.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:20.154+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:20.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:20.154+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:20.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:20.154+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:20.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:20.155+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:21.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:20.722+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:21.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:20.722+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:21.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:20.722+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:21.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:20.723+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:21.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:20.867+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:21.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:20.867+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:21.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:20.867+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:21.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:20.868+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:21.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:21.192+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:21.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:21.192+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:21.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:21.192+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:21.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:21.193+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:22.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:21.686+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:22.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:21.686+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:22.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:21.686+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:22.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:21.687+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:22.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:21.856+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:22.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:21.856+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:22.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:21.856+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:22.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:21.857+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:22.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:22.212+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:22.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:22.212+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:22.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:22.212+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:22.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:22.213+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:23.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:22.719+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:23.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:22.719+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:23.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:22.719+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:23.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:22.720+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:23.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:22.892+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:23.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:22.892+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:23.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:22.892+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:23.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:22.893+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:23.528 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:23.206+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:23.528 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:23.206+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:23.528 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:23.206+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:23.528 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:23.207+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:23.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:23.700+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:23.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:23.700+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:23.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:23.700+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:23.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:23.700+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:24.163 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:23.897+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:24.163 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:23.897+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:24.163 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:23.897+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:24.164 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:23.897+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:24.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:24.163+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:24.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:24.163+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:24.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:24.163+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:24.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:24.164+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:25.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:24.747+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:25.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:24.747+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:25.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:24.747+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:25.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:24.748+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:25.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:24.925+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:25.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:24.925+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:25.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:24.925+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:25.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:24.926+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:25.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:53:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:53:25.196+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 983 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:53:25.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:25.194+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:25.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:25.194+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:25.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:25.194+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:25.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:25.195+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:26.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:25.762+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:26.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:25.762+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:26.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:25.762+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:26.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:25.762+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:26.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:25.944+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:26.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:25.944+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:26.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:25.944+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:26.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:25.945+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:26.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:26.194+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:26.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:26.194+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:26.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:26.194+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:26.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:26.195+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:27.175 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:26.793+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:27.175 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:26.793+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:27.176 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:26.793+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:27.176 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:26.794+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:27.176 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:26.921+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:27.176 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:26.921+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:27.176 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:26.921+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:27.176 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:26.923+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:27.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:27.175+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:27.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:27.175+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:27.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:27.175+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:27.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:27.176+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:28.144 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:27.838+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:28.145 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:27.838+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:28.145 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:27.838+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:28.145 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:27.838+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:28.145 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:27.921+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:28.145 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:27.921+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:28.145 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:27.921+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:28.145 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:27.921+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:28.144+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:28.144+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:28.144+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:28.145+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:29.133 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:28.877+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:29.134 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:28.877+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:29.134 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:28.877+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:29.134 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:28.878+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:29.134 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:28.968+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:29.134 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:28.968+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:29.134 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:28.968+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:29.134 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:28.969+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:29.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:29.133+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:29.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:29.133+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:29.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:29.133+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:29.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:29.134+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:30.109 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:29.921+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:30.109 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:29.921+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:30.109 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:29.921+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:30.109 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:29.922+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:30.110 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:29.964+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:30.110 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:29.964+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:30.110 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:29.964+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:30.110 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:29.965+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:30.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:53:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:53:30.197+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 1011 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:53:30.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:30.109+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:30.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:30.109+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:30.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:30.109+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:30.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:30.109+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:31.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:30.982+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:31.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:30.982+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:31.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:30.982+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:31.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:30.983+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:31.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:30.916+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:31.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:30.916+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:31.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:30.916+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:31.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:30.917+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:31.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:31.115+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:31.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:31.115+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:31.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:31.115+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:31.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:31.116+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:32.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:31.994+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:32.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:31.994+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:32.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:31.994+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:32.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:31.995+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:32.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:32.120+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:32.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:32.120+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:32.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:32.120+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:32.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:32.121+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:32.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:31.906+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:32.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:31.906+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:32.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:31.906+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:32.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:31.907+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:33.156 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:32.888+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:33.156 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:32.888+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:33.156 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:32.888+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:33.156 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:32.889+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:33.156 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:32.967+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:33.156 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:32.967+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:33.156 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:32.967+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:33.157 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:32.968+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:33.528 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:33.155+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:33.529 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:33.155+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:33.529 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:33.155+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:33.529 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:33.156+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:34.200 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:33.916+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:34.201 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:33.916+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:34.201 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:33.916+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:34.201 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:33.917+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:34.201 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:33.938+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:34.201 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:33.938+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:34.201 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:33.938+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:34.201 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:33.938+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:34.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:34.200+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:34.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:34.200+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:34.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:34.200+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:34.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:34.201+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:35.198 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:34.926+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:35.198 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:34.926+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:35.198 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:34.926+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:35.199 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:34.927+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:35.199 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:34.967+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:35.199 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:34.967+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:35.199 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:34.967+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:35.199 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:34.968+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:35.199 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:35.171+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:35.199 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:35.171+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:35.200 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:35.171+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:35.200 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:35.172+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:35.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:53:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:53:35.197+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 1039 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:53:36.273 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:35.991+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:36.273 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:35.991+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:36.273 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:35.991+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:36.273 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:35.992+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:36.274 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:36.125+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:36.274 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:36.125+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:36.274 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:36.125+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:36.274 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:36.126+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:36.274 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:35.928+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:36.275 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:35.928+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:36.275 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:35.928+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:36.275 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:35.929+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:37.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:37.038+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:37.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:37.038+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:37.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:37.038+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:37.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:37.039+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:37.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:37.080+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:37.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:37.080+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:37.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:37.080+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:37.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:37.081+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:37.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:36.936+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:37.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:36.936+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:37.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:36.936+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:37.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:36.936+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:38.014 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:37.920+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:38.014 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:37.920+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:38.014 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:37.920+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:38.014 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:37.921+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:38.014 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:38.011+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:38.015 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:38.011+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:38.015 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:38.012+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:38.015 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:38.013+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:38.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:38.105+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:38.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:38.105+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:38.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:38.105+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:38.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:38.106+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:39.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:39.058+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:39.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:39.058+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:39.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:39.058+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:39.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:39.059+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:39.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:39.066+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:39.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:39.066+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:39.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:39.066+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:39.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:39.067+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:39.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:38.874+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:39.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:38.874+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:39.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:38.874+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:39.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:38.875+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:40.199 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:39.886+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:40.199 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:39.886+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:40.199 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:39.886+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:40.199 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:39.887+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:40.199 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:40.057+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:40.199 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:40.057+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:40.200 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:40.057+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:40.200 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:40.059+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:40.200 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:40.090+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:40.200 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:40.090+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:40.200 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:40.090+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:40.200 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:40.091+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:40.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:53:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:53:40.198+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 1082 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:53:41.139 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:41.050+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:41.139 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:41.050+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:41.139 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:41.050+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:41.139 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:41.051+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:41.140 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:40.847+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:41.140 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:40.847+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:41.140 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:40.847+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:41.140 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:40.848+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:41.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:41.138+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:41.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:41.138+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:41.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:41.138+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:41.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:41.139+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:42.146 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:42.082+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:42.147 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:42.082+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:42.147 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:42.082+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:42.147 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:42.083+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:42.147 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:41.875+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:42.147 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:41.875+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:42.147 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:41.875+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:42.148 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:41.876+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:42.417 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:42.146+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:42.417 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:42.146+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:42.417 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:42.146+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:42.417 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:42.147+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:43.140 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:43.120+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:43.140 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:43.120+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:43.140 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:43.120+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:43.141 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:43.121+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:43.141 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:42.887+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:43.141 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:42.887+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:43.141 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:42.887+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:43.141 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:42.888+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:43.530 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:43.139+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:43.530 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:43.139+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:43.530 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:43.139+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:43.530 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:43.140+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:44.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:43.875+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:44.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:43.875+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:44.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:43.875+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:44.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:43.876+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:44.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:44.099+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:44.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:44.099+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:44.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:44.099+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:44.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:44.100+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:44.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:44.120+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:44.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:44.120+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:44.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:44.120+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:44.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:44.121+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:45.184 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:45.111+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:45.184 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:45.111+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:45.185 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:45.111+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:45.185 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:45.112+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:45.185 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:44.885+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:45.185 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:44.885+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:45.185 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:44.885+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:45.185 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:44.885+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:45.185 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:45.091+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:45.186 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:45.091+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:45.186 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:45.091+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:45.186 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:45.092+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:45.488 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:53:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:53:45.199+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 1127 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:53:46.115 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:46.114+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:46.115 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:46.114+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:46.115 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:46.114+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:46.115 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:46.115+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:46.116 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:45.865+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:46.116 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:45.865+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:46.116 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:45.865+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:46.116 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:45.865+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:46.116 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:46.084+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:46.116 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:46.084+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:46.116 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:46.084+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:46.117 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:46.085+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:47.143 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:46.884+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:47.143 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:46.884+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:47.143 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:46.884+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:47.143 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:46.885+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:47.144 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:47.119+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:47.144 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:47.119+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:47.144 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:47.119+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:47.144 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:47.119+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:47.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:47.143+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:47.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:47.143+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:47.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:47.143+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:47.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:47.144+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:48.175 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:47.884+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:48.175 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:47.884+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:48.175 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:47.884+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:48.175 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:47.885+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:48.175 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:48.079+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:48.175 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:48.079+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:48.175 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:48.079+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:48.176 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:48.079+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:48.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:48.175+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:48.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:48.175+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:48.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:48.175+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:48.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:48.175+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:49.180 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:48.849+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:49.180 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:48.849+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:49.180 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:48.849+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:49.181 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:48.849+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:49.181 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:49.038+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:49.181 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:49.038+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:49.181 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:49.038+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:49.181 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:49.038+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:49.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:49.181+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:49.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:49.181+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:49.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:49.181+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:49.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:49.181+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:50.153 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:50.050+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:50.154 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:50.050+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:50.154 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:50.050+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:50.154 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:50.051+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:50.154 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:49.836+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:50.154 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:49.836+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:50.154 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:49.836+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:50.154 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:49.837+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:50.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:53:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:53:50.201+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 1172 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:53:50.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:50.154+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:50.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:50.154+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:50.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:50.154+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:50.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:50.155+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:51.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:51.105+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:51.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:51.105+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:51.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:51.105+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:51.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:51.106+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:51.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:50.864+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:51.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:50.864+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:51.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:50.864+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:51.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:50.865+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:51.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:51.008+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:51.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:51.008+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:51.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:51.008+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:51.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:51.009+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:52.146 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:52.005+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:52.146 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:52.005+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:52.147 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:52.005+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:52.147 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:52.006+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:52.147 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:51.863+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:52.147 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:51.863+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:52.147 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:51.863+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:52.147 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:51.865+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:52.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:52.147+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:52.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:52.147+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:52.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:52.147+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:52.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:52.148+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:53.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:53.126+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:53.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:53.126+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:53.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:53.126+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:53.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:53.127+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:53.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:52.878+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:53.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:52.878+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:53.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:52.878+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:53.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:52.880+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:53.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:53.026+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:53.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:53.026+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:53.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:53.026+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:53.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:53.026+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:54.171 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:53.854+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:54.171 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:53.854+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:54.171 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:53.854+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:54.171 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:53.855+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:54.172 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:54.056+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:54.172 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:54.056+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:54.172 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:54.056+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:54.172 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:54.056+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:54.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:54.172+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:54.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:54.172+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:54.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:54.172+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:54.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:54.173+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:55.011 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:54.859+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:55.011 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:54.859+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:55.012 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:54.859+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:55.012 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:54.861+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:55.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:53:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:53:55.201+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 1217 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:53:55.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:55.012+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:55.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:55.012+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:55.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:55.012+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:55.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:55.013+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:55.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:55.184+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:55.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:55.184+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:55.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:55.184+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:55.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:55.184+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:55.956 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:53:55.956+0000 7f73a4f8c700 0 monclient(hunting): authenticate timed out after 300 2024-03-24T09:53:55.957 INFO:teuthology.orchestra.run.smithi022.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-03-24T09:53:56.175 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:55.895+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:56.175 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:55.895+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:56.175 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:55.895+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:56.175 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:55.896+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:56.175 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:56.004+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:56.175 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:56.004+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:56.175 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:56.004+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:56.175 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:56.005+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:56.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:56.175+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:56.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:56.175+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:56.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:56.175+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:56.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:56.176+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:56.769 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -e sha1=21b3bd672163092b4e542f772797fcba02d3d4de -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-03-24T09:53:56.989 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:53:57.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:56.977+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:57.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:56.977+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:57.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:56.977+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:57.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:56.978+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:57.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:56.935+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:57.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:56.935+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:57.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:56.935+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:57.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:56.936+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:57.393 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:57.148+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:57.393 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:57.148+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:57.393 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:57.148+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:57.393 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:57.149+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:58.165 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:57.955+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:58.165 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:57.955+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:58.165 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:57.955+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:58.165 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:57.956+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:58.165 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:57.913+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:58.165 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:57.913+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:58.165 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:57.913+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:58.166 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:57.914+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:58.494 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:58.165+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:58.494 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:58.165+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:58.494 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:58.165+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:58.494 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:58.167+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:59.205 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:58.876+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:59.205 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:58.876+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:59.206 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:58.876+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:53:59.206 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:58.876+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:59.206 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:58.915+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:59.206 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:58.915+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:59.206 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:58.915+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:53:59.206 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:58.916+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:53:59.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:59.206+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:59.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:59.206+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:59.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:59.206+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:53:59.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:53:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:53:59.207+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:00.202 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:59.827+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:00.202 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:59.827+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:00.202 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:59.827+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:00.202 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:53:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:53:59.827+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:00.202 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:59.875+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:00.202 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:59.875+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:00.203 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:59.875+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:00.203 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:53:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:53:59.875+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:00.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:54:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:54:00.202+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 1320 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:54:00.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:00.210+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:00.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:00.210+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:00.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:00.210+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:00.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:00.211+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:01.210 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:00.792+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:01.210 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:00.792+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:01.210 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:00.792+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:01.210 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:00.793+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:01.210 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:00.884+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:01.210 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:00.884+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:01.210 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:00.884+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:01.210 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:00.885+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:01.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:01.210+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:01.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:01.210+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:01.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:01.210+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:01.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:01.211+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:02.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:01.772+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:02.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:01.772+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:02.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:01.772+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:02.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:01.773+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:02.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:01.933+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:02.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:01.933+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:02.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:01.933+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:02.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:01.934+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:02.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:02.238+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:02.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:02.238+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:02.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:02.238+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:02.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:02.239+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:03.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:02.724+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:03.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:02.724+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:03.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:02.724+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:03.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:02.724+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:03.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:02.888+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:03.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:02.888+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:03.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:02.888+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:03.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:02.889+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:03.531 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:03.253+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:03.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:03.253+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:03.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:03.253+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:03.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:03.254+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:03.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:03.767+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:03.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:03.767+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:03.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:03.767+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:03.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:03.767+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:04.230 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:03.895+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:04.230 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:03.895+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:04.230 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:03.895+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:04.230 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:03.896+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:04.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:04.229+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:04.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:04.229+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:04.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:04.229+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:04.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:04.230+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:05.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:04.734+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:05.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:04.734+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:05.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:04.734+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:05.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:04.735+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:05.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:04.862+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:05.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:04.862+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:05.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:04.862+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:05.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:04.864+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:05.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:54:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:54:05.202+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 1412 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:54:05.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:05.250+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:05.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:05.250+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:05.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:05.250+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:05.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:05.251+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:06.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:05.781+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:06.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:05.781+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:06.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:05.781+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:06.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:05.782+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:06.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:05.820+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:06.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:05.820+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:06.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:05.820+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:06.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:05.821+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:06.761 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:06.297+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:06.761 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:06.297+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:06.761 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:06.297+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:06.761 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:06.298+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:07.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:06.760+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:07.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:06.760+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:07.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:06.760+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:07.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:06.761+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:07.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:06.820+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:07.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:06.820+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:07.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:06.820+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:07.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:06.821+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:07.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:07.341+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:07.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:07.341+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:07.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:07.341+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:07.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:07.342+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:08.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:07.802+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:08.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:07.802+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:08.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:07.802+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:08.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:07.802+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:08.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:07.864+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:08.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:07.864+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:08.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:07.864+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:08.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:07.865+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:08.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:08.375+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:08.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:08.375+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:08.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:08.375+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:08.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:08.376+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:09.172 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:08.794+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:09.172 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:08.794+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:09.172 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:08.794+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:09.172 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:08.795+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:09.172 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:08.830+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:09.172 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:08.830+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:09.173 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:08.830+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:09.173 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:08.830+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:09.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:09.379+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:09.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:09.379+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:09.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:09.379+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:09.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:09.380+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:10.203 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:09.789+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:10.203 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:09.789+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:10.204 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:09.789+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:10.204 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:09.790+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:10.204 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:09.838+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:10.204 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:09.838+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:10.204 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:09.838+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:10.204 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:09.838+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:10.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:54:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:54:10.202+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 1505 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:54:10.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:10.341+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:10.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:10.341+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:10.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:10.341+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:10.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:10.342+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:11.220 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:10.794+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:11.220 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:10.794+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:11.220 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:10.794+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:11.220 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:10.795+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:11.221 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:10.800+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:11.221 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:10.800+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:11.221 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:10.800+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:11.221 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:10.801+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:11.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:11.323+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:11.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:11.323+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:11.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:11.323+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:11.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:11.324+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:12.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:11.848+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:12.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:11.848+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:12.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:11.848+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:12.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:11.849+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:12.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:11.811+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:12.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:11.811+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:12.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:11.811+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:12.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:11.812+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:12.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:12.318+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:12.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:12.318+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:12.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:12.318+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:12.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:12.319+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:13.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:12.829+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:13.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:12.829+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:13.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:12.829+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:13.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:12.830+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:13.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:12.790+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:13.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:12.790+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:13.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:12.790+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:13.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:12.791+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:13.744 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:13.316+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:13.744 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:13.316+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:13.744 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:13.316+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:13.745 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:13.317+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:14.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:13.744+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:14.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:13.744+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:14.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:13.744+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:14.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:13.745+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:14.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:13.870+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:14.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:13.870+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:14.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:13.870+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:14.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:13.871+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:14.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:14.299+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:14.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:14.299+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:14.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:14.299+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:14.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:14.300+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:15.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:14.734+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:15.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:14.734+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:15.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:14.734+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:15.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:14.734+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:15.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:14.905+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:15.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:14.905+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:15.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:14.905+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:15.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:14.906+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:15.286 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:54:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:54:15.203+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 1599 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:54:15.754 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:15.286+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:15.754 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:15.286+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:15.754 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:15.286+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:15.755 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:15.286+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:16.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:15.754+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:16.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:15.754+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:16.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:15.754+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:16.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:15.755+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:16.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:15.886+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:16.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:15.886+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:16.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:15.886+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:16.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:15.888+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:16.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:16.245+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:16.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:16.245+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:16.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:16.245+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:16.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:16.246+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:17.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:16.891+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:17.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:16.891+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:17.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:16.891+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:17.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:16.891+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:17.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:16.756+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:17.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:16.756+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:17.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:16.756+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:17.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:16.756+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:17.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:17.272+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:17.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:17.272+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:17.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:17.272+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:17.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:17.273+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:18.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:17.883+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:18.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:17.883+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:18.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:17.883+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:18.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:17.884+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:18.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:17.756+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:18.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:17.756+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:18.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:17.756+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:18.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:17.757+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:18.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:18.239+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:18.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:18.239+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:18.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:18.239+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:18.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:18.240+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:19.267 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:18.803+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:19.268 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:18.803+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:19.268 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:18.803+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:19.268 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:18.804+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:19.268 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:18.846+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:19.268 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:18.846+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:19.269 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:18.846+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:19.269 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:18.848+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:19.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:19.283+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:19.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:19.283+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:19.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:19.283+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:19.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:19.284+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:20.035 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:19.820+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:20.035 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:19.820+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:20.035 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:19.820+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:20.035 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:19.821+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:20.036 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:19.779+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:20.036 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:19.779+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:20.036 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:19.779+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:20.036 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:19.780+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:20.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:54:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:54:20.204+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 1691 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:54:20.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:20.235+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:20.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:20.235+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:20.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:20.235+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:20.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:20.235+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:21.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:20.814+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:21.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:20.814+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:21.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:20.814+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:21.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:20.815+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:21.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:20.769+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:21.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:20.769+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:21.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:20.769+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:21.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:20.770+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:21.531 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:21.279+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:21.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:21.279+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:21.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:21.279+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:21.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:21.279+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:22.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:21.808+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:22.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:21.808+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:22.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:21.808+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:22.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:21.809+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:22.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:21.810+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:22.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:21.810+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:22.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:21.810+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:22.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:21.811+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:22.764 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:22.313+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:22.764 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:22.313+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:22.764 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:22.313+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:22.764 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:22.314+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:23.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:22.806+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:23.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:22.806+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:23.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:22.807+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:23.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:22.808+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:23.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:22.764+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:23.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:22.764+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:23.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:22.764+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:23.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:22.765+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:23.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:23.359+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:23.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:23.359+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:23.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:23.359+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:23.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:23.360+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:24.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:23.783+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:24.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:23.783+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:24.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:23.783+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:24.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:23.783+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:24.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:23.801+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:24.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:23.801+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:24.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:23.801+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:24.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:23.802+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:24.763 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:24.313+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:24.763 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:24.313+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:24.764 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:24.313+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:24.764 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:24.313+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:25.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:24.763+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:25.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:24.763+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:25.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:24.763+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:25.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:24.764+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:25.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:24.805+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:25.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:24.806+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:25.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:24.806+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:25.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:24.806+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:25.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:54:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:54:25.204+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 1788 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:54:25.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:25.351+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:25.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:25.351+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:25.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:25.351+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:25.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:25.353+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:26.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:25.728+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:26.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:25.728+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:26.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:25.728+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:26.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:25.729+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:26.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:25.792+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:26.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:25.792+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:26.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:25.792+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:26.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:25.793+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:26.712 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:26.364+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:26.712 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:26.364+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:26.712 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:26.364+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:26.712 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:26.364+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:27.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:26.712+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:27.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:26.712+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:27.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:26.712+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:27.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:26.712+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:27.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:26.832+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:27.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:26.832+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:27.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:26.832+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:27.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:26.832+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:27.678 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:27.354+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:27.678 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:27.354+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:27.678 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:27.354+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:27.678 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:27.355+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:28.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:27.677+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:28.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:27.678+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:28.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:27.678+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:28.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:27.678+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:28.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:27.802+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:28.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:27.802+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:28.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:27.802+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:28.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:27.803+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:28.657 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:28.358+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:28.658 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:28.358+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:28.658 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:28.358+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:28.658 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:28.359+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:29.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:28.657+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:29.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:28.657+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:29.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:28.657+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:29.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:28.658+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:29.035 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:28.798+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:29.035 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:28.798+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:29.035 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:28.798+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:29.035 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:28.799+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:29.626 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:29.400+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:29.626 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:29.400+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:29.626 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:29.400+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:29.626 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:29.401+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:30.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:29.625+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:30.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:29.625+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:30.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:29.625+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:30.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:29.626+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:30.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:29.793+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:30.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:29.793+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:30.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:29.793+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:30.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:29.793+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:30.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:54:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:54:30.205+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 1882 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:54:30.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:30.361+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:30.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:30.361+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:30.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:30.361+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:30.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:30.362+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:31.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:30.649+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:31.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:30.649+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:31.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:30.649+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:31.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:30.650+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:31.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:30.813+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:31.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:30.813+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:31.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:30.813+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:31.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:30.814+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:31.689 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:31.349+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:31.689 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:31.349+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:31.689 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:31.349+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:31.689 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:31.349+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:32.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:31.688+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:32.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:31.688+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:32.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:31.688+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:32.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:31.689+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:32.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:31.861+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:32.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:31.861+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:32.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:31.861+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:32.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:31.862+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:32.657 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:32.359+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:32.657 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:32.359+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:32.657 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:32.359+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:32.657 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:32.360+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:33.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:32.889+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:33.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:32.889+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:33.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:32.889+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:33.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:32.890+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:33.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:32.656+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:33.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:32.656+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:33.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:32.656+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:33.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:32.657+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:33.661 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:33.394+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:33.661 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:33.394+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:33.661 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:33.394+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:33.661 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:33.395+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:33.936 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:33.660+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:33.936 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:33.660+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:33.936 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:33.660+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:33.936 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:33.662+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:34.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:33.935+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:34.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:33.935+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:34.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:33.935+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:34.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:33.936+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:34.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:34.393+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:34.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:34.393+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:34.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:34.393+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:34.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:34.394+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:34.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:34.636+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:34.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:34.636+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:34.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:34.636+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:34.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:34.637+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:35.206 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:34.950+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:35.207 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:34.951+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:35.207 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:34.951+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:35.207 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:34.951+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:35.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:54:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:54:35.206+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 1978 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:54:35.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:35.369+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:35.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:35.369+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:35.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:35.369+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:35.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:35.370+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:35.933 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:35.631+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:35.933 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:35.631+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:35.933 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:35.631+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:35.933 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:35.632+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:36.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:35.932+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:36.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:35.932+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:36.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:35.932+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:36.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:35.934+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:36.624 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:36.323+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:36.625 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:36.323+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:36.625 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:36.323+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:36.625 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:36.323+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:36.910 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:36.624+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:36.910 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:36.624+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:36.910 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:36.624+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:36.910 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:36.625+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:37.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:36.910+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:37.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:36.910+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:37.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:36.910+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:37.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:36.910+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:37.597 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:37.282+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:37.597 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:37.282+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:37.597 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:37.282+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:37.597 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:37.283+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:37.890 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:37.597+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:37.890 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:37.597+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:37.890 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:37.597+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:37.890 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:37.597+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:38.246 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:37.889+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:38.246 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:37.889+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:38.246 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:37.889+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:38.246 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:37.890+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:38.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:38.245+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:38.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:38.245+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:38.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:38.245+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:38.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:38.246+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:38.875+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:38.875+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:38.875+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:38.876+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:39.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:38.646+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:39.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:38.646+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:39.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:38.646+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:39.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:38.647+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:39.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:39.232+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:39.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:39.232+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:39.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:39.232+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:39.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:39.232+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:40.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:39.624+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:40.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:39.624+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:40.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:39.624+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:40.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:39.625+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:40.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:39.861+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:40.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:39.861+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:40.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:39.861+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:40.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:39.862+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:40.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:54:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:54:40.206+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 2071 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:54:40.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:40.241+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:40.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:40.241+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:40.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:40.241+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:40.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:40.241+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:40.878 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:40.582+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:40.879 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:40.582+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:40.879 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:40.582+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:40.879 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:40.583+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:41.233 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:40.878+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:41.233 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:40.878+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:41.233 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:40.878+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:41.233 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:40.879+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:41.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:41.232+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:41.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:41.232+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:41.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:41.232+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:41.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:41.233+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:41.905 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:41.537+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:41.905 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:41.537+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:41.905 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:41.537+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:41.905 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:41.538+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:42.273 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:41.905+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:42.273 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:41.905+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:42.274 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:41.905+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:42.274 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:41.906+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:42.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:42.273+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:42.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:42.273+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:42.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:42.273+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:42.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:42.274+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:42.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:42.490+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:42.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:42.490+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:42.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:42.490+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:42.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:42.491+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:43.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:42.946+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:43.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:42.946+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:43.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:42.946+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:43.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:42.947+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:43.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:43.284+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:43.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:43.284+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:43.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:43.284+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:43.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:43.285+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:43.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:43.514+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:43.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:43.514+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:43.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:43.514+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:43.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:43.515+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:44.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:43.906+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:44.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:43.906+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:44.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:43.906+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:44.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:43.907+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:44.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:44.522+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:44.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:44.522+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:44.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:44.522+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:44.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:44.523+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:44.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:44.328+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:44.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:44.328+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:44.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:44.328+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:44.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:44.329+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:45.207 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:44.923+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:45.208 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:44.923+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:45.208 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:44.923+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:45.208 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:44.924+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:45.512 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:54:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:54:45.207+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 2168 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:54:45.512 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:45.352+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:45.512 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:45.352+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:45.512 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:45.352+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:45.512 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:45.353+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:45.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:45.511+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:45.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:45.511+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:45.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:45.511+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:45.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:45.513+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:46.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:45.931+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:46.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:45.931+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:46.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:45.931+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:46.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:45.932+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:46.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:46.401+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:46.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:46.401+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:46.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:46.401+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:46.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:46.403+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:46.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:46.468+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:46.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:46.468+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:46.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:46.468+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:46.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:46.469+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:47.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:46.959+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:47.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:46.959+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:47.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:46.959+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:47.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:46.960+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:47.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:47.381+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:47.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:47.381+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:47.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:47.381+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:47.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:47.381+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:47.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:47.464+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:47.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:47.464+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:47.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:47.464+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:47.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:47.465+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:48.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:47.927+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:48.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:47.927+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:48.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:47.927+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:48.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:47.928+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:48.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:48.366+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:48.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:48.366+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:48.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:48.366+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:48.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:48.367+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:48.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:48.483+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:48.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:48.483+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:48.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:48.483+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:48.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:48.484+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:49.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:48.956+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:49.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:48.956+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:49.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:48.956+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:49.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:48.957+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:49.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:49.403+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:49.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:49.403+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:49.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:49.403+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:49.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:49.404+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:49.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:49.445+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:49.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:49.445+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:49.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:49.445+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:49.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:49.447+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:50.208 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:49.973+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:50.208 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:49.973+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:50.208 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:49.973+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:50.209 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:49.974+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:50.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:54:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:54:50.208+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 2260 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:54:50.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:50.415+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:50.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:50.415+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:50.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:50.415+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:50.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:50.416+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:50.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:50.403+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:50.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:50.403+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:50.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:50.403+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:50.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:50.403+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:51.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:51.010+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:51.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:51.010+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:51.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:51.010+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:51.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:51.011+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:51.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:51.439+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:51.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:51.439+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:51.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:51.439+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:51.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:51.440+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:51.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:51.428+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:51.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:51.428+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:51.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:51.428+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:51.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:51.429+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:52.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:52.027+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:52.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:52.027+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:52.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:52.027+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:52.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:52.028+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:52.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:52.403+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:52.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:52.403+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:52.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:52.403+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:52.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:52.403+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:52.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:52.475+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:52.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:52.475+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:52.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:52.476+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:52.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:52.476+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:53.380 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:53.042+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:53.381 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:53.042+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:53.381 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:53.042+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:53.381 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:53.043+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:53.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:53.516+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:53.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:53.516+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:53.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:53.516+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:53.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:53.517+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:53.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:53.380+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:53.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:53.380+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:53.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:53.380+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:53.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:53.381+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:54.348 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:54.081+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:54.348 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:54.081+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:54.348 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:54.081+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:54.348 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:54.082+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:54.604 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:54.504+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:54.604 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:54.504+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:54.604 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:54.504+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:54.605 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:54.505+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:54.605 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:54.348+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:54.605 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:54.348+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:54.605 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:54.348+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:54.605 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:54.348+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:55.331 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:54:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:54:55.209+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 2355 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:54:55.332 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:55.058+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:55.332 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:55.058+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:55.332 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:55.058+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:55.332 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:55.059+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:55.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:55.374+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:55.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:55.374+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:55.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:55.374+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:55.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:55.375+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:55.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:55.469+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:55.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:55.469+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:55.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:55.469+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:55.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:55.470+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:56.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:56.017+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:56.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:56.017+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:56.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:56.017+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:56.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:56.018+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:56.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:56.396+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:56.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:56.396+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:56.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:56.396+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:56.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:56.397+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:56.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:56.502+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:56.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:56.502+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:56.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:56.502+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:56.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:56.503+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:57.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:56.998+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:57.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:56.998+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:57.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:56.998+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:57.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:56.999+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:57.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:57.424+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:57.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:57.424+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:57.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:57.424+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:57.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:57.425+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:57.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:57.543+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:57.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:57.543+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:57.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:57.543+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:57.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:57.544+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:58.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:57.954+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:58.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:57.954+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:58.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:57.954+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:58.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:57.955+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:58.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:58.378+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:58.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:58.378+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:58.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:58.378+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:58.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:58.378+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:58.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:58.541+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:58.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:58.541+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:58.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:58.541+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:58.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:58.542+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:59.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:58.976+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:59.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:58.976+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:59.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:58.976+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:54:59.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:54:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:54:58.977+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:59.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:59.493+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:59.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:59.493+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:59.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:59.493+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:54:59.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:54:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:54:59.494+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:54:59.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:59.417+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:59.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:59.417+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:59.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:59.417+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:54:59.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:54:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:54:59.418+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:00.210 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:00.016+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:00.210 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:00.016+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:00.210 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:00.016+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:00.210 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:00.017+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:00.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:55:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:55:00.209+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 2447 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:55:00.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:00.390+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:00.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:00.390+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:00.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:00.390+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:00.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:00.391+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:00.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:00.452+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:00.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:00.452+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:00.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:00.452+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:00.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:00.453+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:01.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:01.005+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:01.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:01.005+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:01.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:01.005+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:01.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:01.005+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:01.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:01.395+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:01.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:01.395+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:01.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:01.395+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:01.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:01.396+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:01.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:01.488+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:01.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:01.488+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:01.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:01.488+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:01.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:01.490+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:02.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:02.005+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:02.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:02.005+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:02.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:02.005+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:02.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:02.005+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:02.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:02.490+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:02.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:02.490+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:02.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:02.490+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:02.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:02.491+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:02.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:02.442+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:02.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:02.442+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:02.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:02.442+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:02.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:02.443+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:03.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:03.023+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:03.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:03.023+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:03.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:03.023+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:03.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:03.024+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:03.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:03.466+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:03.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:03.467+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:03.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:03.467+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:03.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:03.467+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:03.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:03.481+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:03.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:03.481+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:03.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:03.481+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:03.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:03.481+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:04.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:04.016+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:04.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:04.016+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:04.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:04.016+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:04.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:04.017+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:04.699 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:04.439+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:04.699 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:04.439+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:04.699 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:04.439+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:04.699 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:04.440+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:04.700 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:04.518+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:04.700 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:04.518+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:04.700 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:04.518+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:04.700 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:04.518+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:05.335 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:55:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:55:05.210+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 2544 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:55:05.335 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:05.035+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:05.335 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:05.035+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:05.336 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:05.035+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:05.336 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:05.036+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:05.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:05.420+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:05.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:05.420+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:05.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:05.420+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:05.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:05.421+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:05.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:05.511+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:05.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:05.511+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:05.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:05.511+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:05.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:05.512+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:06.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:06.020+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:06.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:06.020+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:06.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:06.020+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:06.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:06.021+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:06.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:06.401+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:06.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:06.401+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:06.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:06.401+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:06.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:06.401+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:06.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:06.525+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:06.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:06.525+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:06.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:06.525+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:06.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:06.526+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:07.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:06.998+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:07.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:06.998+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:07.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:06.998+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:07.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:06.998+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:07.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:07.517+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:07.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:07.517+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:07.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:07.517+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:07.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:07.518+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:07.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:07.363+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:07.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:07.364+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:07.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:07.364+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:07.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:07.364+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:08.377 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:08.036+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:08.378 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:08.036+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:08.378 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:08.036+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:08.378 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:08.038+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:08.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:08.377+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:08.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:08.377+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:08.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:08.377+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:08.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:08.378+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:08.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:08.561+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:08.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:08.561+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:08.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:08.561+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:08.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:08.561+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:09.415 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:09.067+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:09.415 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:09.067+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:09.415 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:09.067+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:09.415 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:09.068+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:09.746 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:09.415+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:09.746 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:09.415+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:09.747 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:09.415+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:09.747 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:09.416+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:09.747 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:09.543+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:09.747 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:09.543+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:09.747 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:09.543+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:09.747 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:09.543+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:10.334 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:55:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:55:10.211+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 2635 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:55:10.335 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:10.033+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:10.335 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:10.033+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:10.335 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:10.033+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:10.335 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:10.033+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:10.585 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:10.447+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:10.585 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:10.447+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:10.585 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:10.447+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:10.585 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:10.447+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:11.010 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:10.585+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:11.010 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:10.585+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:11.010 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:10.585+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:11.011 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:10.586+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:11.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:11.011+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:11.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:11.011+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:11.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:11.011+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:11.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:11.012+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:11.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:11.537+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:11.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:11.537+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:11.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:11.537+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:11.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:11.538+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:11.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:11.491+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:11.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:11.491+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:11.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:11.491+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:11.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:11.492+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:12.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:11.974+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:12.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:11.974+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:12.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:11.974+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:12.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:11.976+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:12.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:12.502+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:12.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:12.502+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:12.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:12.502+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:12.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:12.503+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:12.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:12.514+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:12.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:12.514+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:12.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:12.514+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:12.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:12.515+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:13.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:13.010+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:13.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:13.010+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:13.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:13.010+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:13.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:13.011+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:13.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:13.455+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:13.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:13.455+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:13.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:13.455+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:13.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:13.456+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:13.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:13.564+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:13.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:13.564+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:13.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:13.564+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:13.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:13.565+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:14.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:13.991+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:14.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:13.991+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:14.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:13.991+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:14.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:13.992+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:14.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:14.530+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:14.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:14.530+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:14.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:14.530+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:14.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:14.531+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:14.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:14.424+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:14.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:14.424+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:14.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:14.424+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:14.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:14.425+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:15.212 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:14.984+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:15.212 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:14.984+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:15.212 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:14.984+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:15.212 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:14.985+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:15.524 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:55:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:55:15.212+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 2729 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:55:15.524 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:15.456+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:15.525 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:15.456+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:15.525 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:15.456+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:15.525 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:15.457+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:15.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:15.525+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:15.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:15.525+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:15.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:15.525+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:15.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:15.526+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:16.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:15.978+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:16.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:15.978+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:16.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:15.978+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:16.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:15.979+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:16.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:16.516+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:16.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:16.516+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:16.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:16.516+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:16.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:16.517+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:16.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:16.417+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:16.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:16.417+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:16.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:16.417+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:16.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:16.418+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:17.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:16.962+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:17.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:16.962+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:17.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:16.962+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:17.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:16.962+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:17.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:17.487+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:17.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:17.487+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:17.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:17.487+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:17.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:17.488+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:17.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:17.436+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:17.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:17.436+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:17.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:17.436+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:17.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:17.437+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:18.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:17.939+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:18.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:17.940+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:18.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:17.940+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:18.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:17.940+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:18.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:18.477+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:18.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:18.477+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:18.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:18.477+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:18.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:18.478+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:18.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:18.443+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:18.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:18.443+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:18.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:18.443+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:18.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:18.443+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:19.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:18.892+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:19.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:18.892+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:19.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:18.892+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:19.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:18.892+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:19.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:19.492+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:19.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:19.492+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:19.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:19.492+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:19.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:19.493+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:19.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:19.484+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:19.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:19.484+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:19.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:19.484+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:19.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:19.485+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:20.212 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:19.908+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:20.213 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:19.909+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:20.213 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:19.909+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:20.213 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:19.910+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:20.527 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:55:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:55:20.213+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 2820 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:55:20.527 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:20.460+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:20.527 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:20.460+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:20.527 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:20.460+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:20.528 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:20.460+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:20.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:20.527+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:20.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:20.527+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:20.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:20.528+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:20.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:20.528+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:21.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:20.904+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:21.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:20.904+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:21.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:20.904+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:21.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:20.904+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:21.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:21.419+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:21.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:21.419+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:21.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:21.419+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:21.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:21.419+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:21.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:21.510+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:21.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:21.510+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:21.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:21.510+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:21.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:21.511+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:22.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:21.891+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:22.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:21.891+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:22.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:21.891+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:22.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:21.891+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:22.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:22.499+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:22.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:22.499+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:22.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:22.499+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:22.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:22.500+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:22.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:22.403+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:22.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:22.403+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:22.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:22.403+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:22.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:22.404+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:23.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:22.864+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:23.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:22.864+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:23.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:22.864+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:23.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:22.864+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:23.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:23.407+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:23.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:23.407+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:23.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:23.407+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:23.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:23.408+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:23.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:23.496+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:23.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:23.496+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:23.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:23.496+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:23.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:23.497+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:24.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:23.858+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:24.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:23.858+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:24.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:23.858+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:24.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:23.859+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:24.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:24.390+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:24.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:24.390+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:24.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:24.390+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:24.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:24.391+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:24.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:24.466+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:24.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:24.466+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:24.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:24.466+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:24.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:24.467+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:25.213 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:24.907+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:25.214 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:24.907+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:25.214 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:24.907+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:25.214 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:24.908+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:25.531 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:55:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:55:25.213+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 2916 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:55:25.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:25.455+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:25.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:25.455+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:25.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:25.455+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:25.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:25.457+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:25.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:25.431+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:25.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:25.431+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:25.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:25.431+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:25.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:25.432+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:26.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:25.932+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:26.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:25.932+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:26.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:25.932+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:26.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:25.933+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:26.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:26.472+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:26.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:26.472+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:26.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:26.472+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:26.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:26.472+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:26.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:26.429+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:26.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:26.429+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:26.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:26.429+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:26.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:26.430+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:27.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:26.916+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:27.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:26.916+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:27.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:26.916+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:27.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:26.917+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:27.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:27.460+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:27.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:27.460+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:27.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:27.460+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:27.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:27.461+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:27.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:27.510+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:27.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:27.510+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:27.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:27.510+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:27.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:27.511+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:28.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:27.903+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:28.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:27.903+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:28.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:27.903+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:28.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:27.903+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:28.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:28.508+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:28.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:28.508+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:28.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:28.508+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:28.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:28.509+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:28.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:28.440+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:28.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:28.440+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:28.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:28.440+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:28.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:28.441+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:29.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:28.946+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:29.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:28.946+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:29.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:28.946+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:29.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:28.947+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:29.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:29.480+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:29.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:29.480+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:29.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:29.480+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:29.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:29.481+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:29.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:29.486+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:29.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:29.486+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:29.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:29.486+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:29.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:29.487+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:30.214 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:29.976+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:30.214 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:29.976+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:30.214 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:29.976+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:30.214 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:29.977+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:30.474 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:55:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:55:30.213+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 3007 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:55:30.474 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:30.460+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:30.474 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:30.460+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:30.475 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:30.460+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:30.475 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:30.461+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:30.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:30.474+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:30.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:30.474+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:30.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:30.474+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:30.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:30.475+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:31.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:31.010+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:31.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:31.010+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:31.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:31.010+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:31.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:31.010+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:31.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:31.522+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:31.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:31.522+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:31.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:31.522+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:31.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:31.523+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:31.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:31.477+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:31.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:31.478+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:31.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:31.478+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:31.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:31.478+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:32.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:31.973+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:32.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:31.973+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:32.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:31.973+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:32.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:31.973+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:32.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:32.536+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:32.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:32.536+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:32.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:32.536+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:32.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:32.537+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:32.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:32.456+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:32.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:32.456+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:32.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:32.456+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:32.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:32.457+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:33.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:33.003+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:33.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:33.003+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:33.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:33.003+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:33.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:33.003+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:33.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:33.456+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:33.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:33.456+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:33.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:33.456+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:33.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:33.457+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:33.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:33.549+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:33.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:33.549+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:33.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:33.549+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:33.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:33.549+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:34.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:33.996+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:34.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:33.996+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:34.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:33.996+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:34.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:33.996+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:34.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:34.501+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:34.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:34.501+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:34.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:34.501+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:34.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:34.502+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:34.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:34.525+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:34.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:34.525+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:34.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:34.525+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:34.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:34.526+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:35.225 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:55:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:55:35.214+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 3101 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:55:35.225 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:34.969+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:35.226 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:34.969+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:35.226 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:34.969+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:35.226 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:34.970+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:35.492 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:35.465+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:35.492 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:35.465+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:35.492 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:35.465+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:35.492 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:35.466+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:35.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:35.491+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:35.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:35.491+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:35.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:35.491+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:35.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:35.492+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:36.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:35.960+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:36.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:35.960+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:36.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:35.960+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:36.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:35.961+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:36.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:36.447+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:36.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:36.447+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:36.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:36.447+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:36.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:36.448+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:36.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:36.490+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:36.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:36.490+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:36.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:36.490+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:36.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:36.491+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:37.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:36.987+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:37.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:36.987+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:37.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:36.988+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:37.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:36.988+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:37.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:37.469+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:37.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:37.469+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:37.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:37.469+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:37.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:37.470+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:37.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:37.539+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:37.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:37.539+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:37.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:37.539+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:37.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:37.540+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:38.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:38.023+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:38.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:38.023+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:38.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:38.023+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:38.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:38.023+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:38.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:38.513+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:38.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:38.513+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:38.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:38.513+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:38.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:38.513+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:38.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:38.476+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:38.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:38.476+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:38.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:38.476+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:38.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:38.477+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:39.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:39.024+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:39.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:39.024+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:39.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:39.024+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:39.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:39.025+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:39.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:39.497+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:39.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:39.497+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:39.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:39.497+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:39.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:39.498+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:39.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:39.456+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:39.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:39.456+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:39.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:39.456+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:39.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:39.456+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:40.215 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:40.005+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:40.215 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:40.005+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:40.216 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:40.005+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:40.216 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:40.006+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:40.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:55:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:55:40.215+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 3192 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:55:40.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:40.412+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:40.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:40.412+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:40.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:40.412+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:40.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:40.413+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:41.010 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:40.540+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:41.011 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:40.540+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:41.011 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:40.540+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:41.011 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:40.541+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:41.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:41.010+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:41.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:41.010+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:41.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:41.010+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:41.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:41.011+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:41.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:41.379+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:41.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:41.379+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:41.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:41.379+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:41.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:41.380+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:41.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:41.584+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:41.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:41.584+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:41.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:41.584+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:41.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:41.585+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:42.392 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:42.034+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:42.392 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:42.034+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:42.392 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:42.034+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:42.393 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:42.035+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:42.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:42.392+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:42.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:42.392+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:42.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:42.392+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:42.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:42.393+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:42.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:42.608+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:42.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:42.608+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:42.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:42.608+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:42.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:42.608+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:43.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:42.995+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:43.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:42.995+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:43.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:42.995+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:43.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:42.997+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:43.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:43.426+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:43.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:43.426+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:43.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:43.426+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:43.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:43.427+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:43.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:43.562+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:43.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:43.562+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:43.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:43.562+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:43.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:43.563+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:44.477 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:44.042+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:44.477 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:44.042+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:44.478 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:44.042+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:44.478 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:44.043+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:44.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:44.552+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:44.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:44.552+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:44.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:44.552+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:44.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:44.553+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:44.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:44.477+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:44.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:44.477+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:44.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:44.477+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:44.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:44.477+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:45.339 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:55:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:55:45.216+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 3288 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:55:45.339 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:45.086+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:45.339 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:45.086+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:45.339 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:45.086+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:45.339 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:45.086+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:45.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:45.584+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:45.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:45.584+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:45.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:45.584+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:45.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:45.585+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:45.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:45.508+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:45.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:45.508+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:45.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:45.508+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:45.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:45.509+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:46.489 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:46.082+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:46.489 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:46.082+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:46.489 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:46.082+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:46.489 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:46.083+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:46.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:46.488+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:46.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:46.488+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:46.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:46.488+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:46.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:46.489+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:46.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:46.571+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:46.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:46.571+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:46.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:46.571+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:46.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:46.572+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:47.508 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:47.101+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:47.508 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:47.101+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:47.508 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:47.101+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:47.508 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:47.101+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:47.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:47.584+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:47.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:47.584+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:47.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:47.584+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:47.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:47.585+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:47.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:47.508+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:47.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:47.508+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:47.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:47.508+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:47.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:47.509+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:48.369 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:48.094+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:48.369 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:48.094+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:48.369 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:48.094+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:48.369 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:48.095+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:48.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:48.608+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:48.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:48.608+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:48.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:48.608+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:48.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:48.609+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:48.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:48.480+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:48.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:48.480+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:48.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:48.480+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:48.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:48.481+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:49.439 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:49.084+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:49.440 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:49.084+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:49.440 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:49.084+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:49.440 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:49.085+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:49.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:49.439+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:49.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:49.439+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:49.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:49.439+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:49.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:49.440+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:49.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:49.625+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:49.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:49.625+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:49.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:49.625+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:49.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:49.626+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:50.129 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:50.057+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:50.129 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:50.057+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:50.130 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:50.057+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:50.130 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:50.058+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:50.452 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:55:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:55:50.217+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 3379 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:55:50.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:50.451+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:50.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:50.451+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:50.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:50.451+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:50.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:50.452+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:50.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:50.617+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:50.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:50.617+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:50.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:50.617+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:50.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:50.618+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:51.441 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:51.067+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:51.441 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:51.067+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:51.441 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:51.067+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:51.441 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:51.068+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:51.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:51.450+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:51.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:51.450+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:51.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:51.450+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:51.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:51.451+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:51.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:51.612+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:51.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:51.612+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:51.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:51.612+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:51.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:51.613+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:52.484 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:52.098+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:52.484 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:52.098+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:52.484 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:52.098+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:52.485 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:52.098+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:52.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:52.483+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:52.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:52.483+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:52.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:52.483+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:52.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:52.484+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:52.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:52.620+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:52.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:52.620+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:52.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:52.620+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:52.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:52.621+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:53.443 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:53.094+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:53.443 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:53.094+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:53.443 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:53.094+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:53.443 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:53.095+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:53.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:53.442+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:53.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:53.442+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:53.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:53.442+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:53.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:53.443+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:53.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:53.671+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:53.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:53.671+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:53.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:53.671+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:53.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:53.671+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:54.433 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:54.048+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:54.433 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:54.048+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:54.434 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:54.048+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:54.434 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:54.049+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:54.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:54.433+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:54.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:54.433+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:54.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:54.433+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:54.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:54.434+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:54.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:54.625+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:54.785 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:54.625+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:54.785 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:54.625+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:54.785 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:54.625+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:55.341 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:55:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:55:55.218+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 3473 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:55:55.341 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:55.066+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:55.341 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:55.066+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:55.341 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:55.066+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:55.342 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:55.067+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:55.604 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:55.424+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:55.604 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:55.424+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:55.604 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:55.424+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:55.604 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:55.425+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:56.029 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:55.603+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:56.030 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:55.603+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:56.030 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:55.603+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:56.030 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:55.604+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:56.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:56.029+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:56.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:56.029+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:56.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:56.029+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:56.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:56.030+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:56.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:56.462+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:56.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:56.462+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:56.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:56.462+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:56.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:56.463+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:56.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:56.628+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:56.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:56.628+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:56.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:56.628+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:56.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:56.629+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:57.507 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:57.044+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:57.507 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:57.044+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:57.507 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:57.044+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:57.507 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:57.045+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:57.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:57.506+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:57.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:57.506+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:57.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:57.506+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:57.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:57.507+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:57.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:57.667+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:57.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:57.667+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:57.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:57.667+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:57.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:57.668+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:58.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:58.055+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:58.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:58.055+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:58.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:58.055+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:58.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:58.056+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:59.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:58.534+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:59.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:58.534+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:59.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:58.534+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:59.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:58.535+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:59.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:58.619+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:59.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:58.619+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:59.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:58.619+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:59.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:58.620+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:59.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:59.059+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:59.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:59.059+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:59.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:59.059+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:55:59.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:55:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:55:59.060+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:59.853 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:59.601+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:59.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:59.601+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:59.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:59.601+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:55:59.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:55:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:55:59.601+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:55:59.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:59.543+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:59.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:59.543+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:59.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:59.543+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:55:59.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:55:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:55:59.544+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:00.219 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:00.082+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:00.219 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:00.082+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:00.219 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:00.082+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:00.219 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:00.083+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:00.508 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:56:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:56:00.218+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 3564 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:56:00.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:00.508+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:00.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:00.508+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:00.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:00.508+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:00.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:00.509+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:00.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:00.584+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:00.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:00.584+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:00.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:00.584+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:00.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:00.585+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:01.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:01.059+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:01.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:01.059+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:01.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:01.059+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:01.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:01.059+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:02.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:01.532+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:02.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:01.532+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:02.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:01.532+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:02.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:01.533+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:02.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:01.594+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:02.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:01.594+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:02.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:01.594+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:02.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:01.595+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:02.523 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:02.046+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:02.524 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:02.046+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:02.524 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:02.046+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:02.524 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:02.047+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:02.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:02.523+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:02.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:02.523+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:02.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:02.523+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:02.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:02.524+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:02.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:02.562+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:02.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:02.562+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:02.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:02.562+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:02.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:02.563+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:03.497 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:03.049+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:03.497 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:03.049+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:03.498 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:03.049+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:03.498 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:03.050+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:03.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:03.497+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:03.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:03.497+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:03.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:03.497+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:03.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:03.498+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:03.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:03.517+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:03.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:03.517+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:03.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:03.517+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:03.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:03.518+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:04.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:04.024+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:04.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:04.024+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:04.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:04.024+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:04.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:04.024+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:04.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:04.479+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:04.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:04.479+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:04.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:04.479+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:04.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:04.480+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:04.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:04.522+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:04.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:04.522+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:04.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:04.522+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:04.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:04.523+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:05.278 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:56:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:56:05.219+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 3661 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:56:05.279 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:05.010+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:05.279 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:05.010+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:05.279 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:05.010+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:05.279 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:05.010+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:05.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:05.448+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:05.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:05.448+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:05.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:05.448+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:05.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:05.449+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:05.974 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:05.543+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:05.975 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:05.543+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:05.975 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:05.543+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:05.975 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:05.545+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:06.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:05.974+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:06.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:05.974+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:06.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:05.974+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:06.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:05.975+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:06.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:06.501+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:06.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:06.501+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:06.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:06.501+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:06.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:06.502+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:06.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:06.455+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:06.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:06.455+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:06.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:06.455+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:06.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:06.456+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:07.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:06.977+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:07.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:06.977+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:07.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:06.977+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:07.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:06.977+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:07.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:07.540+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:07.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:07.540+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:07.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:07.540+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:07.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:07.541+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:07.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:07.408+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:07.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:07.408+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:07.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:07.408+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:07.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:07.408+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:08.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:08.005+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:08.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:08.005+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:08.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:08.005+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:08.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:08.006+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:08.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:08.409+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:08.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:08.409+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:08.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:08.409+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:08.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:08.409+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:08.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:08.533+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:08.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:08.533+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:08.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:08.533+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:08.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:08.534+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:09.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:09.018+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:09.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:09.018+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:09.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:09.018+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:09.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:09.018+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:09.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:09.424+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:09.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:09.424+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:09.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:09.424+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:09.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:09.425+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:09.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:09.544+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:09.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:09.544+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:09.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:09.544+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:09.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:09.545+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:10.220 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:10.068+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:10.220 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:10.068+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:10.221 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:10.068+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:10.221 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:10.068+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:10.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:56:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:56:10.220+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 3753 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:56:10.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:10.448+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:10.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:10.448+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:10.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:10.448+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:10.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:10.449+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:11.019 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:10.543+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:11.019 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:10.543+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:11.019 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:10.543+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:11.019 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:10.544+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:11.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:11.019+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:11.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:11.019+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:11.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:11.019+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:11.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:11.020+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:11.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:11.529+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:11.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:11.529+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:11.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:11.529+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:11.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:11.530+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:11.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:11.416+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:11.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:11.416+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:11.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:11.416+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:11.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:11.417+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:12.418 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:12.039+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:12.418 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:12.039+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:12.418 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:12.039+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:12.418 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:12.040+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:12.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:12.481+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:12.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:12.481+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:12.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:12.481+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:12.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:12.482+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:12.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:12.417+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:12.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:12.417+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:12.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:12.417+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:12.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:12.418+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:13.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:12.991+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:13.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:12.991+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:13.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:12.991+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:13.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:12.992+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:13.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:13.376+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:13.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:13.376+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:13.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:13.376+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:13.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:13.377+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:13.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:13.469+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:13.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:13.469+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:13.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:13.469+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:13.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:13.469+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:14.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:13.982+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:14.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:13.983+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:14.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:13.983+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:14.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:13.984+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:14.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:14.370+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:14.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:14.370+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:14.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:14.370+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:14.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:14.372+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:14.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:14.452+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:14.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:14.452+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:14.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:14.452+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:14.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:14.453+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:15.221 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:14.945+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:15.221 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:14.945+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:15.221 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:14.945+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:15.222 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:14.946+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:15.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:56:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:56:15.221+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 3848 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:56:15.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:15.466+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:15.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:15.466+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:15.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:15.467+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:15.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:15.468+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:15.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:15.338+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:15.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:15.338+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:15.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:15.338+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:15.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:15.339+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:16.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:15.979+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:16.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:15.979+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:16.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:15.979+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:16.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:15.980+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:16.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:16.319+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:16.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:16.319+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:16.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:16.319+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:16.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:16.320+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:16.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:16.450+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:16.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:16.450+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:16.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:16.450+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:16.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:16.451+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:17.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:16.967+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:17.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:16.967+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:17.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:16.967+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:17.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:16.968+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:17.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:17.349+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:17.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:17.349+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:17.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:17.349+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:17.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:17.350+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:17.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:17.467+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:17.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:17.467+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:17.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:17.467+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:17.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:17.468+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:18.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:17.976+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:18.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:17.976+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:18.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:17.976+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:18.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:17.977+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:18.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:18.388+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:18.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:18.388+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:18.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:18.388+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:18.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:18.389+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:18.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:18.504+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:18.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:18.504+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:18.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:18.504+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:18.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:18.505+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:19.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:18.984+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:19.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:18.984+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:19.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:18.984+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:19.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:18.985+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:19.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:19.476+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:19.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:19.476+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:19.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:19.476+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:19.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:19.477+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:19.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:19.377+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:19.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:19.377+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:19.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:19.377+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:19.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:19.378+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:20.222 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:19.952+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:20.223 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:19.952+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:20.223 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:19.952+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:20.223 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:19.952+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:20.499 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:56:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:56:20.222+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 3940 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:56:20.499 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:20.357+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:20.499 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:20.357+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:20.500 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:20.357+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:20.500 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:20.358+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:20.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:20.498+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:20.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:20.499+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:20.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:20.499+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:20.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:20.500+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:21.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:20.953+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:21.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:20.953+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:21.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:20.953+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:21.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:20.954+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:21.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:21.540+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:21.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:21.540+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:21.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:21.540+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:21.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:21.541+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:21.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:21.383+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:21.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:21.383+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:21.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:21.383+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:21.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:21.384+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:22.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:21.909+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:22.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:21.909+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:22.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:21.909+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:22.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:21.910+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:22.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:22.361+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:22.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:22.361+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:22.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:22.361+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:22.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:22.361+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:22.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:22.509+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:22.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:22.509+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:22.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:22.509+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:22.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:22.510+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:23.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:22.891+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:23.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:22.891+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:23.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:22.891+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:23.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:22.892+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:23.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:23.316+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:23.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:23.316+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:23.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:23.316+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:23.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:23.316+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:23.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:23.501+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:23.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:23.501+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:23.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:23.501+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:23.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:23.501+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:24.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:23.883+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:24.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:23.883+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:24.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:23.883+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:24.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:23.884+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:24.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:24.498+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:24.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:24.498+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:24.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:24.498+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:24.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:24.498+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:24.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:24.328+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:24.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:24.328+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:24.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:24.328+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:24.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:24.329+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:25.223 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:24.879+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:25.223 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:24.879+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:25.223 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:24.879+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:25.223 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:24.880+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:25.476 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:56:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:56:25.222+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 4037 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:56:25.476 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:25.297+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:25.476 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:25.297+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:25.476 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:25.297+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:25.477 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:25.298+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:25.477 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:25.452+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:25.477 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:25.452+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:25.477 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:25.452+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:25.477 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:25.453+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:26.265 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:25.848+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:26.265 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:25.848+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:26.265 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:25.848+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:26.265 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:25.849+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:26.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:26.264+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:26.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:26.264+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:26.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:26.264+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:26.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:26.266+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:26.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:26.487+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:26.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:26.487+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:26.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:26.487+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:26.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:26.488+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:27.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:26.863+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:27.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:26.863+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:27.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:26.863+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:27.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:26.864+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:27.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:27.513+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:27.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:27.513+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:27.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:27.513+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:27.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:27.514+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:27.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:27.285+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:27.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:27.285+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:27.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:27.285+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:27.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:27.286+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:28.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:27.904+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:28.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:27.904+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:28.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:27.904+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:28.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:27.905+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:28.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:28.545+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:28.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:28.545+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:28.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:28.545+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:28.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:28.546+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:28.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:28.315+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:28.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:28.315+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:28.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:28.315+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:28.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:28.316+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:29.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:28.924+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:29.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:28.924+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:29.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:28.924+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:29.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:28.925+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:29.588 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:29.328+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:29.588 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:29.328+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:29.588 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:29.328+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:29.588 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:29.328+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:29.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:29.588+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:29.855 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:29.588+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:29.855 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:29.588+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:29.855 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:29.589+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:30.224 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:29.967+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:30.224 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:29.967+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:30.224 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:29.967+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:30.224 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:29.967+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:30.525 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:56:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:56:30.223+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 4129 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:56:30.525 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:30.376+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:30.525 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:30.376+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:30.525 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:30.376+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:30.525 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:30.377+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:30.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:30.595+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:30.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:30.595+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:30.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:30.595+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:30.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:30.595+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:31.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:30.939+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:31.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:30.939+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:31.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:30.939+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:31.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:30.940+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:31.607 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:31.357+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:31.607 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:31.357+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:31.607 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:31.357+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:31.607 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:31.357+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:31.908 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:31.606+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:31.908 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:31.606+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:31.908 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:31.606+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:31.908 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:31.607+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:32.281 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:31.908+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:32.281 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:31.908+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:32.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:31.908+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:32.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:31.909+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:32.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:32.376+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:32.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:32.376+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:32.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:32.376+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:32.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:32.376+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:32.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:32.623+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:32.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:32.623+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:32.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:32.623+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:32.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:32.623+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:33.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:32.951+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:33.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:32.951+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:33.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:32.951+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:33.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:32.951+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:33.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:33.390+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:33.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:33.390+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:33.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:33.390+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:33.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:33.391+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:33.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:33.575+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:33.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:33.575+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:33.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:33.575+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:33.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:33.576+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:34.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:33.924+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:34.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:33.924+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:34.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:33.924+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:34.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:33.925+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:34.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:34.557+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:34.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:34.557+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:34.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:34.557+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:34.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:34.558+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:34.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:34.358+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:34.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:34.358+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:34.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:34.358+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:34.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:34.359+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:35.224 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:34.890+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:35.225 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:34.890+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:35.225 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:34.890+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:35.225 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:34.891+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:35.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:56:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:56:35.225+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 4224 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:56:35.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:35.324+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:35.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:35.324+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:35.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:35.324+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:35.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:35.325+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:35.862 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:35.575+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:35.862 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:35.575+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:35.862 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:35.575+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:35.862 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:35.576+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:36.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:35.862+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:36.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:35.862+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:36.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:35.862+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:36.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:35.863+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:36.624 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:36.344+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:36.625 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:36.344+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:36.625 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:36.344+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:36.625 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:36.345+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:36.896 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:36.625+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:36.896 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:36.625+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:36.896 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:36.625+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:36.896 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:36.626+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:37.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:36.897+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:37.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:36.897+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:37.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:36.897+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:37.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:36.897+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:37.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:37.368+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:37.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:37.368+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:37.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:37.368+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:37.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:37.369+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:37.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:37.603+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:37.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:37.603+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:37.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:37.603+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:37.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:37.603+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:38.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:37.901+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:38.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:37.901+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:38.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:37.901+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:38.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:37.901+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:38.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:38.390+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:38.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:38.390+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:38.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:38.390+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:38.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:38.391+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:38.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:38.605+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:38.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:38.605+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:38.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:38.605+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:38.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:38.606+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:39.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:38.906+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:39.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:38.906+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:39.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:38.906+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:39.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:38.907+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:39.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:39.609+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:39.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:39.609+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:39.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:39.609+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:39.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:39.610+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:39.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:39.398+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:39.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:39.398+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:39.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:39.398+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:39.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:39.399+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:40.225 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:39.907+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:40.225 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:39.907+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:40.226 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:39.907+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:40.226 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:39.908+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:40.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:56:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:56:40.226+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 4320 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:56:40.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:40.403+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:40.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:40.403+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:40.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:40.403+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:40.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:40.404+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:40.903 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:40.580+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:40.903 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:40.580+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:40.903 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:40.580+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:40.904 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:40.581+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:41.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:40.903+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:41.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:40.903+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:41.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:40.903+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:41.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:40.905+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:41.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:41.577+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:41.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:41.577+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:41.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:41.577+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:41.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:41.578+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:41.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:41.357+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:41.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:41.357+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:41.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:41.357+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:41.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:41.357+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:42.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:41.893+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:42.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:41.893+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:42.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:41.893+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:42.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:41.894+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:42.568 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:42.310+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:42.568 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:42.310+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:42.568 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:42.310+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:42.568 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:42.311+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:42.881 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:42.568+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:42.881 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:42.568+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:42.882 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:42.568+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:42.882 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:42.569+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:43.274 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:42.882+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:43.274 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:42.882+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:43.274 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:42.882+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:43.274 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:42.883+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:43.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:43.274+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:43.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:43.274+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:43.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:43.274+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:43.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:43.275+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:43.902 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:43.617+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:43.903 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:43.617+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:43.903 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:43.617+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:43.903 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:43.618+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:44.281 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:43.903+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:44.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:43.903+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:44.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:43.903+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:44.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:43.904+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:44.606 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:44.283+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:44.606 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:44.283+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:44.606 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:44.283+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:44.606 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:44.284+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:44.914 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:44.606+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:44.915 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:44.606+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:44.915 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:44.606+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:44.915 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:44.607+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:45.226 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:44.914+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:45.226 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:44.914+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:45.226 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:44.914+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:45.227 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:44.915+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:45.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:56:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:56:45.226+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 4419 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:56:45.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:45.331+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:45.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:45.331+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:45.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:45.331+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:45.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:45.332+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:45.950 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:45.647+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:45.950 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:45.647+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:45.950 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:45.647+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:45.950 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:45.648+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:46.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:45.949+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:46.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:45.949+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:46.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:45.949+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:46.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:45.950+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:46.669 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:46.310+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:46.669 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:46.310+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:46.669 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:46.310+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:46.669 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:46.310+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:46.999 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:46.668+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:46.999 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:46.668+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:46.999 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:46.668+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:46.999 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:46.669+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:47.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:46.998+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:47.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:46.998+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:47.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:46.998+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:47.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:46.999+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:47.681 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:47.307+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:47.682 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:47.307+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:47.682 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:47.307+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:47.682 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:47.308+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:48.011 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:47.681+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:48.011 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:47.681+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:48.011 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:47.681+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:48.011 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:47.682+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:48.262 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:48.011+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:48.262 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:48.011+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:48.262 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:48.011+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:48.262 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:48.012+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:48.262 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:48.261+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:48.262 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:48.261+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:48.262 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:48.261+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:48.262 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:48.261+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:48.969 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:48.709+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:48.969 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:48.709+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:48.969 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:48.709+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:48.969 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:48.710+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:49.260 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:48.968+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:49.260 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:48.968+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:49.260 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:48.968+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:49.260 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:48.969+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:49.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:49.260+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:49.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:49.260+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:49.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:49.260+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:49.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:49.260+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:50.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:49.713+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:50.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:49.713+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:50.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:49.713+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:50.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:49.714+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:50.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:49.922+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:50.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:49.922+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:50.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:49.922+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:50.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:49.922+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:50.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:56:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:56:50.226+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 4513 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:56:50.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:50.263+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:50.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:50.263+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:50.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:50.263+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:50.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:50.263+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:51.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:50.757+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:51.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:50.757+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:51.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:50.757+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:51.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:50.758+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:51.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:50.943+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:51.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:50.943+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:51.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:50.943+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:51.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:50.944+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:51.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:51.229+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:51.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:51.229+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:51.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:51.229+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:51.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:51.230+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:52.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:51.709+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:52.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:51.709+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:52.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:51.709+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:52.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:51.710+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:52.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:51.907+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:52.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:51.907+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:52.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:51.907+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:52.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:51.907+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:52.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:52.181+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:52.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:52.181+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:52.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:52.181+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:52.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:52.182+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:53.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:52.670+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:53.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:52.670+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:53.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:52.670+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:53.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:52.671+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:53.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:52.877+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:53.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:52.877+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:53.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:52.877+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:53.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:52.878+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:53.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:53.199+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:53.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:53.199+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:53.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:53.199+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:53.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:53.200+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:53.844 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:53.719+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:53.844 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:53.719+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:53.845 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:53.719+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:53.845 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:53.720+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:54.244 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:53.844+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:54.244 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:53.844+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:54.244 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:53.844+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:54.244 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:53.845+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:54.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:54.243+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:54.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:54.243+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:54.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:54.243+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:54.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:54.244+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:55.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:54.672+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:55.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:54.672+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:55.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:54.672+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:55.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:54.673+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:55.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:54.809+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:55.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:54.809+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:55.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:54.809+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:55.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:54.810+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:55.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:56:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:56:55.227+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 4619 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:56:55.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:55.216+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:55.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:55.216+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:55.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:55.216+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:55.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:55.217+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:55.952 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:55.633+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:55.953 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:55.633+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:55.953 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:55.633+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:55.953 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:55.634+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:55.953 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:55.827+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:55.953 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:55.827+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:55.953 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:55.827+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:55.953 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:55.828+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:56.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:56.253+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:56.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:56.253+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:56.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:56.253+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:56.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:56.254+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:57.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:56.633+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:57.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:56.633+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:57.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:56.633+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:57.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:56.634+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:57.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:56.876+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:57.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:56.876+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:57.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:56.876+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:57.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:56.876+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:57.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:57.211+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:57.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:57.211+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:57.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:57.211+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:57.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:57.211+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:58.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:57.682+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:58.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:57.682+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:58.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:57.682+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:58.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:57.683+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:58.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:57.836+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:58.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:57.836+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:58.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:57.836+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:58.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:57.837+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:58.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:58.168+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:58.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:58.168+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:58.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:58.168+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:58.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:58.169+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:59.025 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:58.793+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:59.025 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:58.793+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:59.025 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:58.793+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:56:59.025 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:58.793+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:59.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:58.695+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:59.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:58.695+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:59.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:58.695+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:56:59.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:58.696+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:56:59.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:59.154+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:59.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:59.154+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:59.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:59.154+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:56:59.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:56:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:56:59.155+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:00.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:59.714+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:00.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:59.714+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:00.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:59.714+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:00.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:56:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:56:59.715+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:00.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:59.766+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:00.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:59.766+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:00.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:59.766+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:00.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:56:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:56:59.766+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:00.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:57:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:57:00.228+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 4712 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:57:00.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:00.154+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:00.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:00.154+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:00.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:00.154+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:00.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:00.155+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:01.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:00.667+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:01.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:00.667+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:01.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:00.667+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:01.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:00.668+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:01.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:00.744+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:01.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:00.744+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:01.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:00.744+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:01.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:00.745+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:01.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:01.108+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:01.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:01.108+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:01.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:01.108+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:01.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:01.109+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:02.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:01.686+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:02.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:01.686+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:02.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:01.686+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:02.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:01.687+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:02.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:01.704+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:02.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:01.704+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:02.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:01.704+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:02.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:01.705+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:02.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:02.072+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:02.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:02.072+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:02.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:02.072+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:02.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:02.073+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:03.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:02.644+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:03.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:02.644+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:03.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:02.644+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:03.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:02.645+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:03.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:02.702+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:03.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:02.702+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:03.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:02.702+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:03.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:02.702+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:03.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:03.118+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:03.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:03.118+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:03.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:03.118+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:03.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:03.119+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:04.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:03.732+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:04.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:03.732+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:04.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:03.732+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:04.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:03.733+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:04.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:03.595+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:04.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:03.595+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:04.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:03.595+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:04.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:03.596+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:04.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:04.167+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:04.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:04.167+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:04.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:04.167+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:04.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:04.168+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:05.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:04.777+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:05.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:04.777+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:05.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:04.777+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:05.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:04.779+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:05.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:04.546+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:05.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:04.546+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:05.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:04.546+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:05.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:04.547+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:05.531 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:57:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:57:05.229+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 4819 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:57:05.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:05.150+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:05.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:05.150+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:05.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:05.150+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:05.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:05.151+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:05.877 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:05.791+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:05.877 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:05.791+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:05.877 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:05.791+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:05.878 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:05.791+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:05.878 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:05.561+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:05.878 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:05.561+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:05.878 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:05.561+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:05.878 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:05.562+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:06.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:06.104+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:06.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:06.104+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:06.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:06.104+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:06.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:06.105+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:06.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:06.520+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:06.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:06.520+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:06.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:06.520+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:06.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:06.521+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:06.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:06.743+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:06.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:06.743+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:06.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:06.743+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:06.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:06.744+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:07.521 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:07.144+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:07.521 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:07.144+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:07.521 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:07.144+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:07.521 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:07.146+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:07.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:07.520+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:07.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:07.520+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:07.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:07.520+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:07.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:07.521+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:07.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:07.755+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:07.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:07.755+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:07.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:07.755+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:07.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:07.756+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:08.476 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:08.107+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:08.476 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:08.107+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:08.477 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:08.107+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:08.477 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:08.108+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:08.770 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:08.476+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:08.771 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:08.476+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:08.771 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:08.476+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:08.771 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:08.477+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:09.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:08.770+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:09.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:08.770+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:09.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:08.770+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:09.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:08.771+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:09.453 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:09.080+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:09.453 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:09.080+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:09.453 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:09.080+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:09.453 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:09.080+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:09.733 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:09.452+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:09.733 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:09.452+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:09.733 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:09.452+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:09.733 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:09.454+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:10.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:09.732+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:10.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:09.732+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:10.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:09.732+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:10.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:09.733+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:10.410 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:57:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:57:10.230+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 4912 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:57:10.410 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:10.113+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:10.410 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:10.113+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:10.410 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:10.113+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:10.411 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:10.114+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:10.699 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:10.409+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:10.700 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:10.409+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:10.700 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:10.409+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:10.700 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:10.410+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:11.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:10.698+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:11.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:10.698+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:11.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:10.698+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:11.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:10.699+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:11.381 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:11.130+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:11.381 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:11.130+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:11.381 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:11.130+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:11.381 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:11.130+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:11.725 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:11.381+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:11.725 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:11.381+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:11.725 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:11.381+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:11.725 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:11.382+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:12.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:11.724+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:12.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:11.725+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:12.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:11.725+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:12.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:11.725+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:12.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:12.157+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:12.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:12.157+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:12.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:12.157+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:12.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:12.157+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:12.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:12.389+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:12.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:12.389+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:12.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:12.389+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:12.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:12.390+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:13.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:12.710+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:13.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:12.710+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:13.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:12.710+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:13.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:12.711+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:13.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:13.434+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:13.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:13.434+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:13.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:13.434+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:13.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:13.435+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:13.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:13.193+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:13.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:13.193+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:13.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:13.193+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:13.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:13.193+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:14.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:13.715+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:14.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:13.715+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:14.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:13.715+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:14.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:13.716+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:14.464 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:14.182+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:14.464 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:14.182+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:14.464 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:14.182+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:14.464 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:14.182+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:14.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:14.695+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:14.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:14.695+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:14.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:14.695+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:14.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:14.696+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:14.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:14.464+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:14.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:14.464+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:14.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:14.464+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:14.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:14.465+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:15.507 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:57:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:57:15.231+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 5007 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:57:15.507 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:15.178+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:15.507 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:15.178+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:15.507 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:15.178+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:15.507 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:15.179+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:15.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:15.506+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:15.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:15.506+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:15.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:15.506+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:15.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:15.507+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:15.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:15.745+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:15.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:15.745+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:15.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:15.745+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:15.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:15.746+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:16.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:16.191+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:16.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:16.191+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:16.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:16.191+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:16.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:16.192+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:17.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:16.720+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:17.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:16.720+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:17.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:16.720+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:17.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:16.721+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:17.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:16.550+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:17.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:16.550+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:17.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:16.550+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:17.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:16.551+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:17.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:17.241+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:17.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:17.241+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:17.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:17.241+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:17.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:17.242+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:18.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:17.572+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:18.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:17.572+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:18.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:17.572+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:18.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:17.574+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:18.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:17.761+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:18.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:17.761+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:18.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:17.761+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:18.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:17.762+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:18.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:18.192+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:18.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:18.192+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:18.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:18.192+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:18.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:18.193+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:19.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:18.788+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:19.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:18.788+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:19.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:18.788+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:19.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:18.789+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:19.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:18.571+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:19.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:18.571+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:19.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:18.571+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:19.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:18.572+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:19.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:19.241+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:19.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:19.241+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:19.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:19.241+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:19.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:19.241+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:19.835 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:19.538+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:19.835 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:19.538+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:19.835 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:19.538+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:19.835 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:19.539+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:20.232 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:19.834+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:20.232 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:19.834+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:20.232 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:19.835+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:20.232 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:19.835+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:20.510 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:57:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:57:20.232+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 5105 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:57:20.510 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:20.269+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:20.510 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:20.269+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:20.510 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:20.269+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:20.510 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:20.270+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:20.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:20.509+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:20.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:20.509+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:20.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:20.509+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:20.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:20.510+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:21.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:20.852+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:21.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:20.852+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:21.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:20.852+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:21.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:20.852+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:21.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:21.490+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:21.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:21.490+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:21.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:21.490+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:21.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:21.490+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:21.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:21.302+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:21.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:21.302+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:21.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:21.302+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:21.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:21.303+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:22.273 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:21.810+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:22.273 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:21.810+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:22.273 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:21.810+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:22.273 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:21.811+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:22.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:22.272+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:22.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:22.273+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:22.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:22.273+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:22.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:22.273+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:22.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:22.491+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:22.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:22.491+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:22.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:22.491+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:22.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:22.492+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:23.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:22.801+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:23.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:22.801+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:23.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:22.801+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:23.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:22.802+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:23.600 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:23.306+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:23.601 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:23.306+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:23.601 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:23.306+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:23.601 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:23.307+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:23.601 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:23.469+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:23.601 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:23.469+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:23.601 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:23.469+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:23.601 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:23.470+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:24.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:23.813+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:24.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:23.813+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:24.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:23.813+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:24.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:23.814+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:24.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:24.311+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:24.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:24.311+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:24.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:24.311+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:24.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:24.312+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:24.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:24.499+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:24.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:24.499+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:24.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:24.499+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:24.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:24.500+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:25.233 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:24.862+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:25.233 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:24.862+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:25.233 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:24.862+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:25.233 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:24.863+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:25.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:57:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:57:25.232+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 5222 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:57:25.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:25.299+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:25.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:25.299+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:25.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:25.299+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:25.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:25.300+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:25.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:25.461+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:25.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:25.461+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:25.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:25.461+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:25.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:25.461+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:26.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:25.867+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:26.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:25.867+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:26.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:25.867+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:26.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:25.868+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:26.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:26.311+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:26.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:26.311+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:26.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:26.311+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:26.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:26.312+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:26.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:26.482+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:26.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:26.482+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:26.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:26.482+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:26.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:26.483+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:27.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:26.830+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:27.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:26.830+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:27.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:26.830+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:27.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:26.831+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:27.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:27.318+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:27.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:27.318+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:27.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:27.318+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:27.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:27.319+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:27.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:27.526+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:27.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:27.526+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:27.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:27.526+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:27.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:27.526+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:28.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:27.786+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:28.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:27.786+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:28.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:27.786+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:28.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:27.786+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:28.540 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:28.285+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:28.540 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:28.285+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:28.540 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:28.285+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:28.541 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:28.286+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:28.807 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:28.540+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:28.807 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:28.540+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:28.807 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:28.540+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:28.808 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:28.541+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:29.245 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:28.807+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:29.245 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:28.807+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:29.245 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:28.807+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:29.245 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:28.807+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:29.502 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:29.244+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:29.503 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:29.244+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:29.503 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:29.244+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:29.503 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:29.245+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:29.758 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:29.502+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:29.758 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:29.502+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:29.759 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:29.502+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:29.759 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:29.503+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:30.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:29.758+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:30.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:29.758+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:30.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:29.758+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:30.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:29.759+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:30.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:57:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:57:30.233+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 5329 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:57:30.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:30.456+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:30.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:30.456+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:30.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:30.456+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:30.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:30.457+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:30.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:30.213+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:30.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:30.213+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:30.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:30.213+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:30.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:30.213+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:31.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:30.750+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:31.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:30.750+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:31.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:30.750+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:31.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:30.751+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:31.442 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:31.216+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:31.443 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:31.216+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:31.443 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:31.216+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:31.443 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:31.217+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:31.702 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:31.442+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:31.703 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:31.442+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:31.703 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:31.442+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:31.703 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:31.443+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:32.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:31.702+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:32.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:31.702+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:32.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:31.702+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:32.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:31.703+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:32.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:32.403+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:32.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:32.403+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:32.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:32.403+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:32.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:32.404+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:32.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:32.184+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:32.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:32.184+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:32.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:32.184+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:32.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:32.185+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:33.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:32.750+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:33.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:32.750+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:33.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:32.750+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:33.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:32.750+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:33.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:33.194+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:33.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:33.194+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:33.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:33.194+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:33.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:33.195+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:33.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:33.387+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:33.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:33.387+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:33.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:33.387+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:33.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:33.388+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:34.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:33.715+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:34.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:33.715+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:34.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:33.715+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:34.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:33.716+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:34.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:34.423+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:34.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:34.423+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:34.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:34.423+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:34.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:34.424+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:34.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:34.176+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:34.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:34.176+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:34.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:34.176+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:34.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:34.177+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:35.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:34.736+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:35.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:34.736+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:35.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:34.736+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:35.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:34.738+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:35.456 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:57:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:57:35.234+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 5424 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:57:35.457 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:35.188+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:35.457 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:35.188+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:35.457 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:35.188+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:35.457 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:35.189+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:35.776 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:35.456+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:35.776 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:35.456+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:35.776 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:35.456+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:35.776 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:35.457+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:36.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:35.776+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:36.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:35.776+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:36.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:35.776+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:36.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:35.776+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:36.414 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:36.167+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:36.414 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:36.167+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:36.414 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:36.167+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:36.414 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:36.168+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:36.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:36.413+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:36.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:36.413+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:36.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:36.413+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:36.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:36.414+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:37.171 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:36.783+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:37.172 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:36.783+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:37.172 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:36.783+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:37.172 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:36.784+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:37.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:37.395+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:37.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:37.395+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:37.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:37.395+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:37.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:37.396+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:37.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:37.170+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:37.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:37.170+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:37.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:37.170+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:37.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:37.171+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:38.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:37.768+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:38.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:37.768+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:38.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:37.768+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:38.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:37.768+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:38.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:38.348+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:38.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:38.348+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:38.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:38.348+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:38.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:38.348+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:38.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:38.204+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:38.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:38.204+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:38.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:38.204+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:38.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:38.204+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:38.757+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:38.757+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:38.757+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:38.758+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:39.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:39.319+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:39.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:39.319+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:39.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:39.319+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:39.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:39.320+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:39.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:39.203+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:39.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:39.203+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:39.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:39.203+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:39.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:39.204+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:40.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:39.719+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:40.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:39.719+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:40.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:39.719+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:40.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:39.719+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:40.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:57:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:57:40.235+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 5518 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:57:40.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:40.284+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:40.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:40.284+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:40.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:40.284+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:40.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:40.285+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:40.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:40.202+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:40.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:40.202+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:40.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:40.202+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:40.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:40.202+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:41.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:40.677+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:41.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:40.677+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:41.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:40.677+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:41.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:40.677+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:41.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:41.288+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:41.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:41.288+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:41.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:41.288+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:41.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:41.288+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:41.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:41.204+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:41.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:41.204+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:41.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:41.204+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:41.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:41.205+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:41.986 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:41.648+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:41.987 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:41.648+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:41.987 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:41.648+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:41.987 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:41.649+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:42.281 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:42.206+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:42.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:42.206+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:42.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:42.206+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:42.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:42.206+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:42.698 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:42.337+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:42.698 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:42.337+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:42.698 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:42.337+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:42.698 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:42.338+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:43.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:42.697+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:43.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:42.697+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:43.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:42.697+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:43.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:42.698+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:43.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:43.339+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:43.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:43.339+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:43.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:43.339+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:43.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:43.340+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:43.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:43.256+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:43.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:43.256+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:43.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:43.256+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:43.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:43.257+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:44.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:43.657+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:44.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:43.657+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:44.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:43.657+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:44.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:43.657+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:44.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:44.304+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:44.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:44.304+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:44.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:44.304+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:44.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:44.305+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:44.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:44.258+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:44.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:44.258+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:44.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:44.258+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:44.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:44.259+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:45.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:44.689+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:45.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:44.689+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:45.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:44.689+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:45.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:44.689+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:45.296 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:57:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:57:45.236+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 5615 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:57:45.296 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:45.258+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:45.296 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:45.258+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:45.297 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:45.258+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:45.297 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:45.259+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:45.297 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:45.239+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:45.297 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:45.239+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:45.297 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:45.239+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:45.297 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:45.241+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:46.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:45.730+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:46.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:45.730+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:46.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:45.730+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:46.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:45.731+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:46.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:46.296+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:46.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:46.296+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:46.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:46.296+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:46.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:46.297+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:46.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:46.226+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:46.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:46.226+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:46.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:46.226+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:46.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:46.227+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:47.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:46.732+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:47.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:46.732+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:47.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:46.732+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:47.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:46.733+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:47.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:47.305+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:47.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:47.305+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:47.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:47.305+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:47.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:47.306+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:47.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:47.221+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:47.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:47.221+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:47.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:47.221+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:47.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:47.223+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:48.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:47.734+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:48.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:47.734+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:48.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:47.734+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:48.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:47.735+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:48.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:48.262+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:48.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:48.262+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:48.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:48.262+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:48.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:48.263+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:48.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:48.254+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:48.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:48.254+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:48.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:48.254+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:48.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:48.255+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:49.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:48.731+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:49.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:48.731+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:49.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:48.731+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:49.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:48.732+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:49.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:49.240+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:49.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:49.240+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:49.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:49.240+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:49.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:49.240+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:49.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:49.248+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:49.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:49.248+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:49.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:49.248+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:49.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:49.249+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:50.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:49.744+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:50.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:49.744+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:50.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:49.744+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:50.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:49.745+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:50.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:57:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:57:50.237+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 5709 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:57:50.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:50.250+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:50.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:50.250+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:50.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:50.250+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:50.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:50.251+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:50.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:50.231+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:50.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:50.231+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:50.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:50.231+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:50.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:50.232+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:51.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:50.731+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:51.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:50.731+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:51.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:50.731+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:51.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:50.731+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:51.441 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:51.261+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:51.441 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:51.261+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:51.441 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:51.261+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:51.441 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:51.261+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:51.441 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:51.183+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:51.442 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:51.183+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:51.442 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:51.183+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:51.442 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:51.184+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:52.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:51.690+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:52.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:51.690+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:52.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:51.690+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:52.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:51.690+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:52.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:52.224+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:52.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:52.224+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:52.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:52.224+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:52.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:52.225+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:52.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:52.298+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:52.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:52.298+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:52.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:52.298+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:52.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:52.299+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:53.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:52.674+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:53.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:52.674+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:53.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:52.674+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:53.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:52.674+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:53.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:53.324+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:53.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:53.324+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:53.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:53.324+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:53.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:53.325+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:53.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:53.188+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:53.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:53.188+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:53.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:53.188+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:53.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:53.189+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:54.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:53.626+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:54.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:53.626+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:54.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:53.626+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:54.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:53.628+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:54.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:54.205+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:54.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:54.205+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:54.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:54.205+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:54.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:54.205+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:54.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:54.296+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:54.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:54.296+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:54.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:54.296+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:54.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:54.297+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:55.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:54.615+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:55.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:54.615+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:55.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:54.615+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:55.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:54.616+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:55.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:57:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:57:55.239+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 5804 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:57:55.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:55.192+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:55.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:55.192+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:55.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:55.192+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:55.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:55.193+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:55.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:55.322+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:55.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:55.322+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:55.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:55.322+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:55.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:55.323+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:56.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:55.602+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:56.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:55.602+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:56.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:55.602+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:56.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:55.602+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:56.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:56.333+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:56.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:56.333+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:56.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:56.333+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:56.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:56.334+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:56.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:56.154+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:56.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:56.154+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:56.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:56.154+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:56.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:56.154+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:57.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:56.588+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:57.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:56.588+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:57.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:56.588+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:57.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:56.589+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:57.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:57.136+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:57.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:57.136+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:57.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:57.136+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:57.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:57.137+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:57.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:57.308+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:57.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:57.308+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:57.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:57.308+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:57.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:57.308+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:58.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:57.560+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:58.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:57.560+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:58.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:57.560+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:58.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:57.561+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:58.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:58.146+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:58.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:58.146+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:58.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:58.146+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:58.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:58.147+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:58.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:58.298+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:58.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:58.298+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:58.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:58.298+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:58.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:58.299+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:59.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:58.566+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:59.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:58.566+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:59.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:58.566+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:59.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:58.567+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:59.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:59.107+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:59.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:59.107+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:59.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:59.107+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:57:59.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:57:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:57:59.108+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:59.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:59.275+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:59.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:59.275+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:59.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:59.275+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:57:59.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:57:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:57:59.276+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:57:59.853 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:59.576+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:59.853 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:59.576+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:59.853 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:59.576+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:57:59.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:57:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:57:59.578+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:00.239 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:00.077+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:00.239 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:00.077+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:00.239 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:00.077+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:00.239 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:00.078+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:00.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:58:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:58:00.240+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 5898 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:58:00.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:00.271+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:00.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:00.271+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:00.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:00.271+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:00.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:00.272+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:01.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:00.539+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:01.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:00.539+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:01.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:00.539+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:01.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:00.540+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:01.420 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:01.104+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:01.421 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:01.104+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:01.421 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:01.104+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:01.421 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:01.105+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:01.421 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:01.283+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:01.421 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:01.283+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:01.421 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:01.283+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:01.421 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:01.284+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:01.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:01.553+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:01.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:01.553+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:01.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:01.553+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:01.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:01.554+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:02.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:02.078+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:02.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:02.078+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:02.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:02.078+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:02.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:02.079+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:02.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:02.263+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:02.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:02.263+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:02.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:02.263+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:02.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:02.264+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:03.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:02.585+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:03.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:02.585+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:03.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:02.585+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:03.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:02.586+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:03.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:03.252+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:03.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:03.252+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:03.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:03.252+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:03.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:03.253+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:03.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:03.079+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:03.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:03.079+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:03.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:03.079+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:03.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:03.080+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:04.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:03.595+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:04.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:03.595+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:04.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:03.595+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:04.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:03.596+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:04.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:04.253+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:04.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:04.253+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:04.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:04.253+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:04.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:04.254+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:04.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:04.079+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:04.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:04.079+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:04.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:04.079+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:04.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:04.080+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:05.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:04.608+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:05.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:04.608+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:05.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:04.608+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:05.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:04.609+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:05.362 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:58:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:58:05.241+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 5995 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:58:05.362 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:05.077+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:05.362 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:05.077+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:05.362 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:05.077+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:05.363 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:05.078+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:05.363 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:05.207+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:05.363 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:05.207+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:05.363 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:05.207+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:05.363 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:05.208+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:06.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:05.636+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:06.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:05.636+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:06.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:05.636+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:06.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:05.637+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:06.467 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:06.114+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:06.468 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:06.114+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:06.468 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:06.114+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:06.468 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:06.115+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:06.468 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:06.224+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:06.468 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:06.224+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:06.468 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:06.224+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:06.468 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:06.225+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:06.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:06.640+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:06.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:06.640+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:06.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:06.640+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:06.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:06.641+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:07.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:07.240+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:07.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:07.240+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:07.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:07.240+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:07.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:07.241+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:07.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:07.122+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:07.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:07.122+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:07.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:07.122+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:07.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:07.123+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:08.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:07.639+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:08.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:07.639+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:08.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:07.639+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:08.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:07.640+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:08.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:08.256+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:08.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:08.256+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:08.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:08.256+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:08.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:08.257+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:08.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:08.112+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:08.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:08.112+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:08.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:08.112+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:08.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:08.113+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:09.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:08.685+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:09.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:08.685+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:09.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:08.685+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:09.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:08.686+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:09.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:09.253+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:09.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:09.253+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:09.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:09.253+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:09.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:09.254+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:09.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:09.153+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:09.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:09.153+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:09.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:09.153+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:09.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:09.154+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:10.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:09.660+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:10.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:09.660+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:10.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:09.660+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:10.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:09.660+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:10.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:58:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:58:10.241+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 6089 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:58:10.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:10.222+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:10.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:10.222+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:10.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:10.222+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:10.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:10.222+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:10.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:10.178+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:10.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:10.178+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:10.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:10.178+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:10.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:10.179+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:11.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:10.707+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:11.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:10.707+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:11.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:10.707+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:11.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:10.707+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:11.515 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:11.185+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:11.515 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:11.185+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:11.515 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:11.185+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:11.515 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:11.186+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:11.515 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:11.226+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:11.515 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:11.226+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:11.515 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:11.226+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:11.515 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:11.227+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:11.781 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:11.681+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:11.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:11.681+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:11.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:11.681+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:11.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:11.682+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:12.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:12.233+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:12.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:12.233+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:12.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:12.233+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:12.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:12.234+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:12.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:12.259+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:12.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:12.259+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:12.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:12.259+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:12.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:12.260+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:13.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:12.652+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:13.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:12.652+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:13.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:12.652+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:13.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:12.652+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:13.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:13.250+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:13.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:13.250+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:13.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:13.250+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:13.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:13.251+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:13.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:13.232+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:13.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:13.232+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:13.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:13.232+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:13.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:13.233+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:14.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:13.689+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:14.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:13.689+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:14.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:13.689+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:14.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:13.691+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:14.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:14.238+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:14.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:14.238+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:14.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:14.238+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:14.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:14.240+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:14.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:14.200+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:14.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:14.200+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:14.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:14.200+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:14.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:14.201+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:15.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:14.709+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:15.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:14.709+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:15.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:14.709+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:15.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:14.710+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:15.363 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:58:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:58:15.242+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 6184 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:58:15.363 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:15.233+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:15.364 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:15.233+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:15.364 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:15.233+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:15.364 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:15.234+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:15.364 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:15.261+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:15.364 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:15.261+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:15.364 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:15.261+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:15.364 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:15.261+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:16.016 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:15.703+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:16.016 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:15.703+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:16.016 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:15.703+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:16.016 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:15.704+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:16.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:16.294+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:16.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:16.294+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:16.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:16.294+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:16.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:16.294+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:16.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:16.281+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:16.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:16.281+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:16.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:16.281+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:16.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:16.282+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:17.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:16.667+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:17.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:16.667+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:17.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:16.667+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:17.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:16.668+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:17.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:17.233+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:17.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:17.233+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:17.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:17.233+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:17.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:17.233+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:17.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:17.330+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:17.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:17.330+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:17.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:17.330+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:17.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:17.331+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:18.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:17.668+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:18.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:17.668+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:18.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:17.668+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:18.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:17.669+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:18.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:18.352+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:18.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:18.352+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:18.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:18.352+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:18.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:18.353+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:18.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:18.210+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:18.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:18.210+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:18.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:18.210+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:18.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:18.211+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:19.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:18.650+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:19.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:18.650+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:19.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:18.650+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:19.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:18.650+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:19.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:19.207+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:19.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:19.207+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:19.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:19.207+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:19.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:19.208+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:19.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:19.382+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:19.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:19.382+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:19.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:19.382+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:19.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:19.383+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:20.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:19.623+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:20.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:19.623+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:20.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:19.623+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:20.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:19.624+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:20.425 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:58:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:58:20.242+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 6278 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:58:20.425 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:20.169+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:20.425 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:20.169+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:20.426 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:20.169+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:20.426 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:20.170+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:20.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:20.592+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:20.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:20.592+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:20.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:20.592+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:20.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:20.593+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:20.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:20.425+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:20.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:20.425+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:20.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:20.425+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:20.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:20.426+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:21.392 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:21.122+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:21.392 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:21.122+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:21.392 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:21.122+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:21.392 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:21.122+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:21.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:21.391+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:21.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:21.391+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:21.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:21.391+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:21.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:21.392+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:21.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:21.615+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:21.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:21.615+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:21.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:21.615+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:21.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:21.616+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:22.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:22.377+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:22.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:22.377+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:22.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:22.377+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:22.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:22.378+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:22.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:22.159+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:22.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:22.159+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:22.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:22.159+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:22.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:22.160+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:23.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:22.664+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:23.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:22.664+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:23.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:22.664+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:23.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:22.664+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:23.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:23.364+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:23.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:23.365+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:23.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:23.365+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:23.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:23.365+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:23.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:23.208+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:23.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:23.208+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:23.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:23.208+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:23.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:23.209+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:24.031 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:23.657+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:24.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:23.657+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:24.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:23.657+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:24.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:23.657+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:24.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:24.370+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:24.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:24.370+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:24.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:24.370+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:24.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:24.371+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:24.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:24.240+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:24.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:24.240+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:24.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:24.240+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:24.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:24.241+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:25.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:24.687+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:25.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:24.687+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:25.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:24.687+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:25.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:24.688+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:25.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:58:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:58:25.243+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 6375 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:58:25.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:25.360+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:25.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:25.360+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:25.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:25.360+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:25.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:25.361+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:25.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:25.259+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:25.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:25.259+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:25.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:25.259+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:25.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:25.260+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:26.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:25.715+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:26.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:25.715+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:26.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:25.715+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:26.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:25.716+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:26.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:26.212+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:26.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:26.212+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:26.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:26.212+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:26.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:26.213+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:26.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:26.319+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:26.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:26.319+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:26.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:26.319+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:26.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:26.320+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:27.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:26.691+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:27.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:26.691+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:27.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:26.691+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:27.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:26.692+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:27.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:27.351+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:27.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:27.351+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:27.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:27.351+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:27.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:27.352+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:27.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:27.190+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:27.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:27.190+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:27.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:27.190+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:27.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:27.191+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:28.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:27.685+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:28.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:27.685+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:28.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:27.685+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:28.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:27.685+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:28.378+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:28.378+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:28.378+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:28.379+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:28.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:28.156+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:28.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:28.156+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:28.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:28.156+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:28.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:28.156+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:29.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:28.717+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:29.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:28.717+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:29.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:28.717+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:29.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:28.718+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:29.402 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:29.110+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:29.402 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:29.110+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:29.403 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:29.110+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:29.403 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:29.111+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:29.754 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:29.402+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:29.754 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:29.402+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:29.754 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:29.402+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:29.754 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:29.403+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:30.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:29.753+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:30.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:29.753+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:30.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:29.753+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:30.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:29.755+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:30.408 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:58:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:58:30.244+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 6469 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:58:30.409 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:30.140+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:30.409 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:30.140+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:30.409 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:30.140+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:30.409 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:30.141+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:30.724 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:30.408+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:30.724 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:30.408+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:30.724 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:30.408+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:30.724 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:30.409+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:31.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:30.724+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:31.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:30.724+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:31.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:30.724+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:31.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:30.725+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:31.451 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:31.184+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:31.452 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:31.184+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:31.452 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:31.184+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:31.452 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:31.185+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:31.717 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:31.451+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:31.717 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:31.451+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:31.717 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:31.451+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:31.717 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:31.452+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:31.718 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:31.695+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:31.718 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:31.695+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:31.718 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:31.695+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:31.718 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:31.696+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:32.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:32.403+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:32.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:32.403+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:32.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:32.403+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:32.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:32.404+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:32.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:32.187+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:32.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:32.187+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:32.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:32.187+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:32.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:32.188+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:33.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:32.692+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:33.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:32.692+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:33.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:32.692+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:33.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:32.693+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:33.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:33.394+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:33.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:33.394+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:33.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:33.394+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:33.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:33.395+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:33.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:33.228+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:33.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:33.228+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:33.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:33.228+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:33.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:33.229+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:34.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:33.691+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:34.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:33.691+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:34.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:33.691+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:34.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:33.692+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:34.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:34.424+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:34.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:34.424+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:34.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:34.424+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:34.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:34.425+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:34.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:34.230+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:34.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:34.230+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:34.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:34.230+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:34.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:34.231+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:35.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:34.685+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:35.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:34.685+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:35.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:34.685+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:35.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:34.686+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:35.531 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:58:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:58:35.245+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 6564 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:58:35.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:35.397+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:35.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:35.397+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:35.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:35.397+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:35.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:35.398+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:35.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:35.209+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:35.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:35.209+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:35.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:35.209+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:35.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:35.209+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:36.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:35.669+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:36.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:35.669+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:36.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:35.669+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:36.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:35.670+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:36.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:36.422+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:36.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:36.422+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:36.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:36.422+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:36.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:36.423+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:36.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:36.188+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:36.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:36.188+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:36.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:36.188+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:36.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:36.189+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:37.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:36.621+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:37.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:36.621+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:37.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:36.621+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:37.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:36.621+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:37.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:37.225+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:37.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:37.225+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:37.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:37.225+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:37.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:37.225+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:37.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:37.457+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:37.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:37.457+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:37.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:37.457+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:37.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:37.459+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:38.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:37.584+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:38.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:37.584+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:38.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:37.584+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:38.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:37.585+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:38.457 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:38.185+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:38.458 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:38.185+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:38.458 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:38.185+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:38.458 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:38.186+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:38.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:38.457+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:38.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:38.457+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:38.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:38.457+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:38.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:38.458+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:38.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:38.635+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:38.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:38.635+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:38.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:38.635+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:38.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:38.635+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:39.493 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:39.229+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:39.493 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:39.229+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:39.493 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:39.229+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:39.493 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:39.229+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:39.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:39.492+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:39.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:39.492+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:39.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:39.492+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:39.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:39.493+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:39.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:39.676+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:39.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:39.676+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:39.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:39.676+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:39.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:39.677+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:40.501 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:58:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:58:40.247+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 6658 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:58:40.501 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:40.209+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:40.501 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:40.209+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:40.501 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:40.209+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:40.501 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:40.210+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:40.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:40.628+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:40.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:40.628+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:40.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:40.628+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:40.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:40.628+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:40.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:40.500+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:40.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:40.500+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:40.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:40.500+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:40.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:40.501+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:41.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:41.227+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:41.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:41.227+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:41.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:41.227+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:41.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:41.227+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:41.817 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:41.536+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:41.817 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:41.536+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:41.817 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:41.536+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:41.817 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:41.536+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:41.817 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:41.658+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:41.817 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:41.658+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:41.817 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:41.658+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:41.818 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:41.659+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:42.489 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:42.198+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:42.490 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:42.198+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:42.490 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:42.198+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:42.490 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:42.199+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:42.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:42.489+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:42.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:42.489+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:42.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:42.489+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:42.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:42.490+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:42.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:42.651+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:42.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:42.651+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:42.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:42.651+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:42.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:42.652+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:43.530 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:43.247+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:43.530 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:43.247+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:43.530 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:43.247+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:43.530 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:43.248+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:43.781 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:43.642+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:43.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:43.642+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:43.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:43.642+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:43.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:43.643+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:43.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:43.529+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:43.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:43.529+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:43.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:43.529+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:43.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:43.530+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:44.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:44.274+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:44.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:44.274+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:44.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:44.274+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:44.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:44.275+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:45.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:44.574+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.095487+0000 front 2024-03-24T09:48:20.095435+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:45.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:44.574+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.095502+0000 front 2024-03-24T09:48:20.095467+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:45.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:44.574+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.095522+0000 front 2024-03-24T09:48:20.095533+0000 (oldest deadline 2024-03-24T09:48:43.595539+0000) 2024-03-24T09:58:45.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:44.574+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:45.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:44.623+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:45.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:44.623+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:45.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:44.623+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:45.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:44.624+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:45.370 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:58:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:58:45.247+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 6755 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:58:45.371 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:45.259+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:45.371 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:45.259+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:45.371 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:45.259+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:45.371 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:45.260+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:45.781 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:45.595+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:45.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:45.600+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:45.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:45.600+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:45.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:45.600+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:45.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:45.601+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:46.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:46.262+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:46.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:46.262+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:46.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:46.262+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:46.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:46.263+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:46.865 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:46.613+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:46.866 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:46.632+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:46.866 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:46.632+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:46.866 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:46.632+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:46.866 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:46.633+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:47.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:47.252+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:47.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:47.252+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:47.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:47.252+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:47.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:47.252+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:48.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:47.604+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:48.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:47.608+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:48.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:47.608+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:48.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:47.608+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:48.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:47.608+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:48.531 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:48.231+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:48.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:48.231+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:48.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:48.231+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:48.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:48.232+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:49.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:48.602+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:49.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:48.644+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:49.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:48.644+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:49.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:48.644+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:49.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:48.645+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:49.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:49.212+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:49.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:49.212+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:49.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:49.212+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:49.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:49.213+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:49.853 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:49.575+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:49.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:49.677+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:49.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:49.677+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:49.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:49.677+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:49.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:49.678+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:50.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:58:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:58:50.248+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 6849 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:58:50.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:50.259+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:50.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:50.259+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:50.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:50.259+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:50.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:50.259+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:51.031 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:50.570+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:51.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:50.629+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:20.403657+0000 front 2024-03-24T09:48:20.403635+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:51.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:50.629+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:20.403533+0000 front 2024-03-24T09:48:20.403620+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:51.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:50.629+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:20.403578+0000 front 2024-03-24T09:48:20.403592+0000 (oldest deadline 2024-03-24T09:48:44.503418+0000) 2024-03-24T09:58:51.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:50.630+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:51.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:51.236+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:51.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:51.236+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:51.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:51.236+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:51.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:51.237+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:51.911 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:51.593+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:51.911 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:51.589+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:52.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:52.255+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:52.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:52.255+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:52.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:52.255+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:52.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:52.256+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:52.880 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:52.550+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:52.881 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:52.585+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:53.519 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:53.214+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:53.519 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:53.214+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:53.519 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:53.214+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:53.519 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:53.214+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:53.781 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:53.518+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:53.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:53.543+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:54.521 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:54.234+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:54.521 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:54.234+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:54.521 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:54.234+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:54.521 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:54.235+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:54.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:54.521+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:54.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:54.547+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:55.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:58:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:58:55.249+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 6944 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:58:55.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:55.273+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:55.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:55.273+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:55.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:55.273+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:55.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:55.274+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:55.952 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:55.569+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:55.952 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:55.590+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:56.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:56.259+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:56.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:56.259+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:56.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:56.259+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:56.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:56.260+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:56.960 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:56.590+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:56.961 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:56.623+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:57.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:57.271+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 since back 2024-03-24T09:48:19.639263+0000 front 2024-03-24T09:48:19.639421+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:57.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:57.271+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 since back 2024-03-24T09:48:19.639448+0000 front 2024-03-24T09:48:19.639512+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:57.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:57.271+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 since back 2024-03-24T09:48:19.639473+0000 front 2024-03-24T09:48:19.639500+0000 (oldest deadline 2024-03-24T09:48:45.539356+0000) 2024-03-24T09:58:57.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:57.272+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:58.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:57.554+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:58.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:57.595+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:58.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:58.258+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:58.713 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T09:58:58.712+0000 7fab36696700 0 monclient(hunting): authenticate timed out after 300 2024-03-24T09:58:58.714 INFO:teuthology.orchestra.run.smithi022.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-03-24T09:58:59.025 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:58.577+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:59.025 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:58.603+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:59.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:58:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:58:59.212+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:59.623 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -e sha1=21b3bd672163092b4e542f772797fcba02d3d4de -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-03-24T09:58:59.834 INFO:teuthology.orchestra.run.smithi022.stderr:Inferring config /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/mon.smithi022/config 2024-03-24T09:58:59.855 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:58:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:58:59.616+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:58:59.855 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:58:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:58:59.613+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:00.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:59:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:59:00.251+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 7037 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:59:00.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:00.200+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:00.976 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:00.598+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:00.976 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:00.664+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:01.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:01.198+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:02.008 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:01.608+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:02.008 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:01.667+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:02.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:02.246+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:03.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:02.641+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:03.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:02.634+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:03.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:03.249+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:04.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:03.592+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:04.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:03.672+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:04.531 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:04.254+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:05.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:04.598+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:05.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:04.625+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:05.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:59:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:59:05.252+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 7133 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:59:05.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:05.228+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:06.031 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:05.639+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:06.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:05.587+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:06.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:06.219+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:07.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:06.645+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:07.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:06.597+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:07.531 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:07.227+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:08.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:07.680+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:08.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:07.639+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:08.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:08.251+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:09.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:08.634+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:09.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:08.702+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:09.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:08.702+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:09.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:08.702+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:09.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:08.702+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:09.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:09.224+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:10.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:09.614+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:10.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:09.670+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:10.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:09.670+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:10.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:09.670+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:10.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:09.670+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:10.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:59:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:59:10.253+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 7226 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:59:10.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:10.183+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:11.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:10.591+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:11.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:10.717+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:11.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:10.717+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:11.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:10.717+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:11.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:10.718+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:11.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:11.146+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:12.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:11.548+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:12.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:11.728+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:12.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:11.728+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:12.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:11.728+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:12.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:11.730+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:12.530 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:12.151+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:12.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:12.530+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:12.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:12.716+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:12.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:12.716+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:12.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:12.716+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:12.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:12.717+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:13.526 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:13.170+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:13.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:13.527+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:13.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:13.527+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:13.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:13.527+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:13.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:13.527+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:13.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:13.762+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:13.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:13.762+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:13.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:13.762+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:13.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:13.762+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:14.478 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:14.191+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:14.780 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:14.479+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:14.780 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:14.479+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:14.781 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:14.479+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:14.781 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:14.479+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:15.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:14.780+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:15.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:14.780+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:15.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:14.780+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:15.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:14.781+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:15.379 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:59:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:59:15.254+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 7320 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:59:15.379 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:15.196+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:15.767 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:15.466+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:15.767 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:15.466+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:15.767 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:15.466+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:15.768 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:15.467+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:16.031 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:15.767+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:16.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:15.767+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:16.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:15.767+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:16.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:15.768+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:16.332 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:16.193+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:16.737 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:16.450+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:16.737 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:16.450+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:16.737 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:16.450+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:16.737 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:16.451+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:17.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:16.737+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:17.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:16.737+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:17.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:16.737+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:17.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:16.738+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:17.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:17.192+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:17.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:17.411+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:17.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:17.411+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:17.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:17.411+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:17.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:17.411+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:18.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:17.779+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:18.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:17.779+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:18.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:17.779+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:18.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:17.780+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:18.459 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:18.166+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:18.760 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:18.459+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:18.760 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:18.459+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:18.760 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:18.459+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:18.760 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:18.461+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:19.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:18.760+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:19.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:18.760+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:19.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:18.760+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:19.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:18.761+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:19.456 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:19.183+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:19.752 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:19.456+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:19.752 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:19.456+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:19.752 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:19.456+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:19.752 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:19.457+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:20.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:19.753+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:20.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:19.753+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:20.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:19.753+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:20.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:19.754+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:20.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:59:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:59:20.256+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 7413 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:59:20.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:20.213+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:20.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:20.213+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:20.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:20.213+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:20.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:20.213+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:20.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:20.419+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:20.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:20.419+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:20.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:20.419+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:20.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:20.420+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:21.217 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:20.785+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:21.217 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:20.785+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:21.218 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:20.785+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:21.218 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:20.786+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:21.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:21.384+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:21.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:21.384+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:21.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:21.384+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:21.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:21.384+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:21.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:21.218+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:21.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:21.218+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:21.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:21.218+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:21.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:21.219+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:22.195 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:21.798+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:22.195 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:21.798+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:22.196 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:21.798+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:22.196 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:21.799+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:22.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:22.196+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:22.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:22.196+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:22.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:22.196+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:22.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:22.197+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:22.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:22.394+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:22.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:22.394+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:22.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:22.394+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:22.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:22.394+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:23.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:22.772+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:23.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:22.772+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:23.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:22.772+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:23.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:22.772+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:23.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:23.160+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:23.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:23.160+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:23.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:23.160+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:23.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:23.161+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:23.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:23.394+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:23.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:23.394+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:23.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:23.394+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:23.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:23.395+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:24.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:23.790+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:24.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:23.790+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:24.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:23.790+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:24.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:23.790+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:24.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:24.396+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:24.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:24.396+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:24.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:24.396+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:24.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:24.397+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:24.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:24.167+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:24.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:24.167+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:24.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:24.167+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:24.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:24.168+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:25.169 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:24.783+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:25.169 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:24.783+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:25.169 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:24.783+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:25.169 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:24.783+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:25.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:59:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:59:25.257+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 7510 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:59:25.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:25.395+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:25.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:25.395+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:25.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:25.395+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:25.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:25.397+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:25.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:25.169+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:25.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:25.169+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:25.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:25.169+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:25.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:25.170+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:26.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:25.736+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:26.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:25.736+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:26.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:25.736+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:26.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:25.736+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:26.410 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:26.157+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:26.410 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:26.157+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:26.410 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:26.157+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:26.410 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:26.157+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:26.695 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:26.411+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:26.696 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:26.411+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:26.696 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:26.411+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:26.696 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:26.411+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:27.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:26.696+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:27.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:26.696+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:27.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:26.696+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:27.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:26.697+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:27.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:27.427+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:27.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:27.427+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:27.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:27.427+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:27.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:27.428+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:27.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:27.187+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:27.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:27.187+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:27.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:27.187+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:27.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:27.187+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:28.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:27.718+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:28.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:27.718+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:28.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:27.718+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:28.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:27.718+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:28.469 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:28.209+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:28.469 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:28.209+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:28.470 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:28.209+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:28.470 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:28.210+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:28.720 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:28.470+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:28.720 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:28.470+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:28.720 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:28.470+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:28.720 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:28.471+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:29.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:28.720+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:29.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:28.720+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:29.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:28.720+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:29.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:28.721+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:29.485 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:29.170+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:29.485 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:29.170+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:29.486 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:29.170+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:29.486 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:29.171+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:29.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:29.485+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:29.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:29.485+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:29.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:29.485+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:29.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:29.486+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:29.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:29.681+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:29.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:29.681+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:29.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:29.681+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:29.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:29.681+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:30.506 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:59:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:59:30.257+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 7604 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:59:30.507 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:30.152+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:30.507 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:30.153+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:30.507 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:30.153+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:30.507 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:30.154+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:30.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:30.727+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:30.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:30.727+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:30.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:30.727+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:30.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:30.728+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:30.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:30.505+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:30.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:30.505+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:30.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:30.505+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:30.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:30.506+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:31.529 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:31.134+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:31.529 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:31.134+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:31.529 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:31.134+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:31.529 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:31.136+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:31.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:31.734+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:31.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:31.734+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:31.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:31.734+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:31.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:31.734+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:31.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:31.529+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:31.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:31.529+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:31.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:31.529+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:31.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:31.529+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:32.509 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:32.109+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:32.509 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:32.109+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:32.509 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:32.109+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:32.509 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:32.110+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:32.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:32.508+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:32.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:32.508+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:32.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:32.508+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:32.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:32.509+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:33.156 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:32.783+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:33.156 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:32.783+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:33.156 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:32.783+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:33.156 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:32.784+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:33.505 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:33.156+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:33.505 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:33.156+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:33.505 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:33.156+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:33.505 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:33.157+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:33.777 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:33.504+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:33.777 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:33.504+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:33.777 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:33.504+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:33.777 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:33.505+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:34.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:33.776+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:34.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:33.776+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:34.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:33.776+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:34.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:33.777+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:34.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:34.162+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:34.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:34.162+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:34.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:34.162+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:34.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:34.163+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:35.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:34.799+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:35.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:34.799+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:35.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:34.799+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:35.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:34.800+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:35.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:34.550+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:35.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:34.550+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:35.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:34.550+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:35.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:34.551+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:35.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:59:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:59:35.258+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 7698 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:59:35.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:35.156+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:35.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:35.156+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:35.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:35.156+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:35.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:35.156+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:35.888 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:35.553+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:35.888 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:35.553+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:35.889 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:35.553+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:35.889 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:35.554+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:35.889 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:35.790+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:35.889 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:35.790+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:35.889 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:35.790+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:35.889 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:35.791+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:36.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:36.120+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:36.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:36.120+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:36.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:36.120+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:36.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:36.121+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:37.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:36.548+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:37.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:36.548+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:37.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:36.548+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:37.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:36.549+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:37.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:36.776+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:37.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:36.776+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:37.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:36.776+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:37.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:36.777+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:37.375 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:37.078+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:37.375 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:37.078+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:37.375 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:37.078+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:37.375 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:37.078+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:37.748 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:37.590+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:37.748 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:37.590+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:37.748 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:37.590+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:37.748 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:37.591+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:38.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:37.747+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:38.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:37.747+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:38.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:37.747+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:38.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:37.748+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:38.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:38.070+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:38.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:38.070+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:38.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:38.070+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:38.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:38.071+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:38.616+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:38.616+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:38.616+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:38.617+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:39.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:38.752+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:39.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:38.752+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:39.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:38.752+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:39.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:38.752+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:39.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:39.114+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:39.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:39.114+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:39.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:39.114+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:39.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:39.115+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:39.853 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:39.773+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:39.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:39.773+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:39.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:39.773+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:39.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:39.773+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:39.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:39.570+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:39.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:39.570+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:39.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:39.570+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:39.855 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:39.570+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:40.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:59:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:59:40.259+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 7791 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:59:40.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:40.164+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:40.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:40.164+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:40.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:40.164+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:40.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:40.165+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:41.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:40.619+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:41.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:40.619+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:41.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:40.619+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:41.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:40.620+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:41.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:40.780+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:41.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:40.780+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:41.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:40.780+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:41.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:40.781+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:41.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:41.124+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:41.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:41.124+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:41.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:41.124+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:41.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:41.124+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:42.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:41.631+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:42.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:41.631+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:42.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:41.631+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:42.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:41.632+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:42.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:41.820+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:42.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:41.820+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:42.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:41.820+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:42.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:41.820+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:42.436 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:42.077+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:42.436 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:42.077+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:42.436 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:42.077+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:42.436 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:42.078+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:42.781 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:42.611+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:42.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:42.611+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:42.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:42.611+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:42.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:42.612+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:43.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:43.071+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:43.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:43.071+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:43.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:43.071+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:43.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:43.072+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:43.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:42.869+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:43.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:42.869+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:43.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:42.869+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:43.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:42.869+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:43.863 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:43.586+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:43.863 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:43.586+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:43.863 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:43.586+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:43.864 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:43.586+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:44.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:43.863+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:44.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:43.863+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:44.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:43.863+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:44.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:43.863+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:44.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:44.108+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:44.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:44.108+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:44.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:44.108+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:44.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:44.109+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:44.832 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:44.548+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:44.832 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:44.548+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:44.832 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:44.548+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:44.832 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:44.549+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:45.104 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:44.831+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:45.104 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:44.831+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:45.105 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:44.831+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:45.105 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:44.832+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:45.105 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:45.062+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:45.105 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:45.062+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:45.105 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:45.062+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:45.105 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:45.063+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:45.384 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:59:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:59:45.260+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 7887 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:59:45.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:45.584+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:45.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:45.584+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:45.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:45.584+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:45.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:45.584+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:46.118 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:45.821+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:46.118 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:45.821+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:46.118 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:45.821+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:46.119 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:45.822+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:46.119 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:46.059+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:46.119 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:46.059+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:46.119 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:46.059+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:46.119 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:46.059+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:46.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:46.550+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:46.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:46.550+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:46.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:46.550+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:46.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:46.551+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:47.092 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:46.793+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:47.093 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:46.793+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:47.093 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:46.793+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:47.093 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:46.794+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:47.426 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:47.092+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:47.426 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:47.092+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:47.426 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:47.092+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:47.426 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:47.093+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:47.764 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:47.763+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:47.764 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:47.763+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:47.764 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:47.763+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:47.764 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:47.521+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:47.765 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:47.521+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:47.765 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:47.521+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:47.765 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:47.522+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:48.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:47.764+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:48.528 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:48.065+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:48.528 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:48.065+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:48.529 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:48.065+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:48.529 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:48.066+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:48.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:48.742+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:48.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:48.742+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:48.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:48.742+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:48.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:48.743+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:48.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:48.528+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:48.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:48.528+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:48.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:48.528+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:48.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:48.529+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:49.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:49.085+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:49.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:49.085+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:49.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:49.085+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:49.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:49.086+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:49.853 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:49.765+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:49.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:49.765+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:49.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:49.765+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:49.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:49.766+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:49.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:49.535+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:49.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:49.535+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:49.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:49.535+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:49.855 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:49.536+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:50.262 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:50.045+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:50.262 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:50.045+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:50.262 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:50.045+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:50.262 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:50.046+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:50.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:59:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:59:50.261+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 7981 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:59:50.815 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:50.534+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:50.815 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:50.534+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:50.815 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:50.534+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:50.815 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:50.534+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:51.249 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:50.815+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:51.249 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:50.815+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:51.249 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:50.815+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:51.249 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:50.816+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:51.249 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:51.014+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:51.249 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:51.014+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:51.250 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:51.014+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:51.250 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:51.015+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:51.839 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:51.555+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:51.839 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:51.555+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:51.839 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:51.555+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:51.839 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:51.556+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:52.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:51.838+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:52.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:51.838+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:52.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:51.838+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:52.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:51.839+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:52.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:52.049+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:52.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:52.049+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:52.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:52.049+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:52.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:52.050+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:52.843 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:52.544+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:52.843 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:52.544+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:52.843 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:52.544+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:52.843 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:52.545+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:53.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:52.843+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:53.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:52.843+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:53.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:52.843+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:53.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:52.844+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:53.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:53.042+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:53.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:53.042+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:53.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:53.042+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:53.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:53.042+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:53.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:53.517+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:53.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:53.517+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:53.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:53.517+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:53.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:53.518+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:54.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:54.085+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:54.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:54.085+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:54.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:54.085+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:54.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:54.086+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:54.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:53.862+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:54.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:53.862+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:54.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:53.862+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:54.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:53.862+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:54.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:54.530+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:54.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:54.530+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:54.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:54.530+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:54.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:54.530+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:55.107 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:54.822+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:55.107 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:54.822+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:55.107 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:54.822+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:55.107 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:54.822+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:55.385 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 09:59:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T09:59:55.262+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 8075 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T09:59:55.386 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:55.106+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:55.386 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:55.106+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:55.386 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:55.106+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:55.386 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:55.107+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:55.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:55.516+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:55.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:55.516+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:55.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:55.516+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:55.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:55.517+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:56.155 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:55.848+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:56.155 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:55.848+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:56.156 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:55.848+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:56.156 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:55.849+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:56.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:56.155+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:56.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:56.155+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:56.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:56.155+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:56.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:56.156+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:56.803 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:56.539+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:56.804 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:56.539+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:56.804 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:56.539+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:56.804 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:56.540+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:57.185 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:56.803+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:57.185 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:56.803+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:57.185 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:56.803+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:57.185 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:56.804+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:57.485 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:57.185+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:57.485 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:57.185+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:57.485 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:57.185+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:57.485 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:57.186+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:57.780 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:57.499+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:57.780 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:57.499+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:57.781 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:57.499+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:57.781 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:57.500+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:58.031 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:57.780+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:58.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:57.780+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:58.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:57.780+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:58.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:57.781+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:58.480 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:58.214+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:58.481 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:58.214+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:58.481 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:58.214+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:58.481 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:58.215+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:58.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:58.480+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:58.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:58.480+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:58.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:58.480+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:58.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:58.481+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:59.192 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:58.801+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:59.192 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:58.801+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:59.192 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:58.801+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T09:59:59.192 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:58.802+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:59.530 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:59.191+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:59.530 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:59.191+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:59.531 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:59.191+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T09:59:59.531 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 09:59:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T09:59:59.193+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T09:59:59.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:59.530+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:59.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:59.530+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:59.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:59.530+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T09:59:59.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 09:59:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T09:59:59.531+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:00.232 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:59.841+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:00.232 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:59.841+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:00.232 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:59.841+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:00.232 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 09:59:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T09:59:59.843+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:00.525 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:00:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:00:00.263+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 8167 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:00:00.525 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:00.231+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:00.525 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:00.231+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:00.525 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:00.231+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:00.525 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:00.232+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:00.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:00.524+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:00.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:00.524+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:00.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:00.524+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:00.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:00.525+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:01.272 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:00.880+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:01.272 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:00.880+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:01.272 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:00.880+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:01.272 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:00.880+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:01.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:01.271+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:01.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:01.271+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:01.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:01.271+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:01.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:01.272+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:01.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:01.484+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:01.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:01.484+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:01.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:01.484+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:01.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:01.485+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:02.239 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:01.869+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:02.239 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:01.869+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:02.239 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:01.869+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:02.240 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:01.870+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:02.517 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:02.239+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:02.517 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:02.239+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:02.517 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:02.239+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:02.517 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:02.240+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:02.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:02.516+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:02.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:02.516+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:02.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:02.516+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:02.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:02.517+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:03.234 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:02.827+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:03.234 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:02.827+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:03.234 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:02.827+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:03.234 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:02.828+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:03.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:03.233+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:03.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:03.233+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:03.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:03.233+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:03.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:03.234+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:03.817 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:03.560+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:03.817 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:03.560+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:03.817 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:03.560+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:03.817 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:03.561+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:04.271 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:03.816+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:04.272 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:03.816+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:04.272 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:03.816+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:04.272 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:03.817+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:04.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:04.271+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:04.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:04.271+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:04.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:04.271+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:04.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:04.272+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:04.859 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:04.588+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:04.859 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:04.588+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:04.859 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:04.588+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:04.859 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:04.588+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:05.253 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:04.859+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:05.254 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:04.859+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:05.254 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:04.859+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:05.254 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:04.859+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:05.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:00:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:00:05.264+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 8263 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:00:05.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:05.253+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:05.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:05.253+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:05.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:05.253+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:05.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:05.254+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:06.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:05.586+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:06.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:05.586+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:06.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:05.586+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:06.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:05.586+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:06.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:05.814+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:06.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:05.814+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:06.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:05.814+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:06.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:05.815+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:06.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:06.235+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:06.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:06.235+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:06.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:06.235+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:06.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:06.236+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:06.846 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:06.555+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:06.847 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:06.555+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:06.847 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:06.555+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:06.847 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:06.555+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:07.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:06.846+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:07.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:06.846+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:07.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:06.846+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:07.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:06.847+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:07.593 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:07.285+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:07.593 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:07.285+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:07.593 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:07.285+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:07.593 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:07.286+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:07.895 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:07.592+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:07.895 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:07.592+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:07.895 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:07.592+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:07.895 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:07.593+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:08.265 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:07.894+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:08.265 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:07.894+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:08.265 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:07.894+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:08.265 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:07.895+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:08.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:08.265+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:08.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:08.265+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:08.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:08.265+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:08.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:08.265+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:08.893 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:08.561+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:08.893 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:08.561+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:08.893 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:08.561+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:08.893 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:08.562+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:09.223 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:08.892+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:09.223 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:08.892+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:09.223 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:08.892+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:09.223 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:08.893+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:09.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:09.223+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:09.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:09.223+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:09.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:09.223+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:09.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:09.223+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:09.853 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:09.535+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:09.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:09.535+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:09.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:09.535+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:09.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:09.536+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:10.195 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:09.904+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:10.195 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:09.904+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:10.195 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:09.904+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:10.195 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:09.904+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:10.498 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:00:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:00:10.266+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 8354 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:00:10.498 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:10.195+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:10.498 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:10.195+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:10.498 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:10.195+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:10.498 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:10.195+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:10.781 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:10.497+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:10.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:10.497+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:10.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:10.497+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:10.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:10.499+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:11.235 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:10.926+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:11.235 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:10.926+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:11.235 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:10.926+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:11.235 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:10.927+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:11.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:11.234+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:11.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:11.234+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:11.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:11.234+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:11.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:11.235+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:11.922 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:11.539+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:11.922 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:11.539+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:11.922 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:11.539+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:11.922 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:11.540+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:12.209 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:11.922+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:12.210 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:11.922+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:12.210 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:11.922+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:12.210 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:11.922+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:12.531 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:12.209+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:12.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:12.209+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:12.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:12.209+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:12.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:12.211+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:12.963 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:12.545+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:12.963 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:12.545+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:12.964 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:12.546+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:12.964 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:12.546+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:13.281 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:13.181+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:13.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:13.181+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:13.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:13.181+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:13.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:13.182+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:13.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:12.963+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:13.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:12.963+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:13.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:12.963+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:13.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:12.964+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:13.969 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:13.542+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:13.969 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:13.542+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:13.970 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:13.542+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:13.970 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:13.543+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:14.227 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:13.969+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:14.227 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:13.969+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:14.227 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:13.969+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:14.227 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:13.969+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:14.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:14.227+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:14.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:14.227+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:14.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:14.227+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:14.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:14.228+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:14.942 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:14.587+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:14.942 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:14.587+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:14.943 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:14.587+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:14.943 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:14.588+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:15.199 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:14.942+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:15.199 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:14.942+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:15.199 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:14.942+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:15.199 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:14.942+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:15.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:00:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:00:15.267+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 8448 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:00:15.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:15.199+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:15.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:15.199+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:15.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:15.199+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:15.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:15.200+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:15.824 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:15.600+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:15.824 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:15.600+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:15.824 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:15.600+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:15.824 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:15.601+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:16.245 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:15.951+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:16.245 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:15.951+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:16.245 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:15.951+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:16.245 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:15.952+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:16.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:16.244+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:16.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:16.244+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:16.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:16.244+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:16.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:16.245+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:16.982 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:16.637+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:16.982 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:16.637+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:16.982 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:16.637+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:16.982 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:16.638+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:17.249 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:16.981+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:17.250 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:16.981+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:17.250 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:16.981+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:17.250 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:16.982+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:17.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:17.249+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:17.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:17.249+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:17.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:17.249+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:17.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:17.250+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:18.014 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:17.686+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:18.014 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:17.686+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:18.014 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:17.686+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:18.014 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:17.686+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:18.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:18.013+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:18.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:18.014+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:18.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:18.014+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:18.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:18.014+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:18.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:18.261+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:18.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:18.261+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:18.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:18.261+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:18.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:18.262+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:18.965 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:18.656+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:18.965 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:18.656+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:18.966 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:18.656+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:18.966 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:18.657+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:19.261 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:18.964+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:19.261 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:18.965+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:19.261 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:18.965+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:19.261 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:18.966+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:19.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:19.260+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:19.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:19.260+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:19.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:19.260+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:19.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:19.261+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:20.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:19.944+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:20.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:19.944+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:20.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:19.944+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:20.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:19.945+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:20.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:19.702+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:20.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:19.702+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:20.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:19.702+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:20.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:19.703+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:20.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:00:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:00:20.268+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 8540 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:00:20.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:20.221+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:20.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:20.221+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:20.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:20.221+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:20.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:20.222+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:20.958 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:20.686+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:20.958 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:20.686+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:20.958 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:20.686+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:20.958 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:20.687+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:21.217 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:20.957+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:21.217 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:20.957+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:21.218 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:20.957+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:21.218 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:20.958+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:21.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:21.217+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:21.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:21.217+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:21.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:21.217+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:21.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:21.218+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:21.991 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:21.733+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:21.991 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:21.733+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:21.991 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:21.733+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:21.991 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:21.734+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:22.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:21.990+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:22.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:21.990+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:22.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:21.990+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:22.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:21.991+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:22.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:22.202+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:22.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:22.202+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:22.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:22.202+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:22.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:22.202+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:23.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:22.956+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:23.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:22.956+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:23.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:22.956+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:23.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:22.957+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:23.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:22.732+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:23.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:22.732+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:23.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:22.732+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:23.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:22.733+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:23.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:23.221+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:23.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:23.221+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:23.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:23.221+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:23.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:23.221+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:23.974 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:23.770+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:23.974 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:23.770+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:23.975 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:23.770+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:23.975 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:23.771+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:24.258 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:23.974+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:24.258 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:23.974+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:24.258 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:23.974+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:24.258 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:23.975+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:24.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:24.258+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:24.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:24.258+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:24.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:24.258+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:24.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:24.258+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:25.270 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:24.812+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:25.270 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:24.812+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:25.271 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:24.812+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:25.271 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:24.813+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:25.271 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:25.022+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:25.271 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:25.022+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:25.271 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:25.023+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:25.271 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:25.023+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:25.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:00:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:00:25.270+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 8637 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:00:25.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:25.290+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:25.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:25.290+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:25.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:25.291+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:25.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:25.291+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:26.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:25.859+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:26.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:25.859+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:26.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:25.859+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:26.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:25.860+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:26.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:25.995+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:26.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:25.995+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:26.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:25.995+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:26.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:25.996+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:26.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:26.330+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:26.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:26.330+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:26.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:26.330+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:26.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:26.331+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:27.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:26.814+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:27.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:26.814+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:27.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:26.814+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:27.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:26.814+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:27.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:27.040+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:27.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:27.040+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:27.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:27.040+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:27.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:27.040+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:27.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:27.373+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:27.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:27.373+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:27.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:27.373+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:27.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:27.374+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:28.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:27.784+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:28.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:27.784+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:28.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:27.784+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:28.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:27.784+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:28.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:27.995+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:28.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:27.995+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:28.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:27.995+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:28.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:27.996+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:28.747 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:28.406+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:28.748 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:28.406+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:28.748 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:28.406+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:28.748 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:28.407+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:29.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:28.991+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:29.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:28.991+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:29.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:28.991+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:29.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:28.991+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:29.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:28.747+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:29.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:28.747+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:29.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:28.747+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:29.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:28.748+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:29.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:29.360+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:29.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:29.360+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:29.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:29.360+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:29.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:29.361+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:30.271 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:29.793+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:30.272 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:29.793+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:30.272 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:29.793+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:30.272 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:29.794+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:30.272 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:29.968+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:30.272 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:29.968+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:30.273 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:29.968+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:30.273 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:29.969+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:30.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:00:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:00:30.271+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 8730 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:00:30.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:30.353+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:30.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:30.353+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:30.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:30.353+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:30.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:30.354+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:31.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:31.017+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:31.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:31.017+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:31.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:31.017+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:31.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:31.018+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:31.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:30.824+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:31.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:30.824+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:31.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:30.824+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:31.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:30.825+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:31.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:31.396+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:31.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:31.396+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:31.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:31.396+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:31.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:31.397+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:32.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:32.061+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:32.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:32.061+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:32.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:32.061+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:32.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:32.062+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:32.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:31.820+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:32.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:31.820+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:32.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:31.820+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:32.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:31.821+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:32.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:32.369+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:32.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:32.369+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:32.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:32.369+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:32.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:32.369+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:33.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:33.014+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:33.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:33.014+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:33.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:33.014+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:33.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:33.015+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:33.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:32.789+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:33.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:32.789+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:33.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:32.789+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:33.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:32.790+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:33.743 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:33.403+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:33.743 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:33.403+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:33.743 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:33.403+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:33.743 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:33.404+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:34.004 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:33.742+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:34.004 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:33.742+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:34.004 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:33.742+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:34.004 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:33.743+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:34.256 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:34.003+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:34.256 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:34.003+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:34.256 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:34.003+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:34.257 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:34.004+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:34.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:34.368+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:34.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:34.368+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:34.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:34.368+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:34.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:34.369+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:35.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:34.750+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:35.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:34.750+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:35.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:34.750+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:35.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:34.751+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:35.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:34.959+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:35.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:34.959+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:35.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:34.959+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:35.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:34.959+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:35.531 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:00:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:00:35.272+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 8825 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:00:35.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:35.350+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:35.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:35.350+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:35.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:35.350+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:35.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:35.351+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:36.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:35.780+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:36.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:35.780+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:36.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:35.780+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:36.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:35.781+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:36.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:35.937+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:36.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:35.937+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:36.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:35.937+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:36.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:35.938+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:36.770 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:36.388+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:36.770 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:36.388+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:36.770 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:36.388+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:36.770 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:36.389+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:37.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:36.978+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:37.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:36.978+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:37.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:36.978+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:37.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:36.979+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:37.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:36.771+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:37.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:36.771+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:37.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:36.771+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:37.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:36.771+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:37.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:37.402+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:37.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:37.402+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:37.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:37.402+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:37.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:37.403+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:38.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:38.015+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:38.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:38.015+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:38.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:38.015+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:38.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:38.015+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:38.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:37.798+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:38.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:37.798+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:38.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:37.798+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:38.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:37.800+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:38.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:38.399+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:38.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:38.399+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:38.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:38.399+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:38.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:38.399+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:39.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:39.001+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:39.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:39.001+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:39.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:39.001+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:39.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:39.002+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:39.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:38.840+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:39.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:38.840+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:39.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:38.840+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:39.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:38.841+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:39.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:39.428+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:39.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:39.428+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:39.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:39.428+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:39.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:39.429+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:40.274 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:40.012+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:40.274 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:40.012+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:40.274 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:40.012+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:40.274 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:40.013+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:40.274 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:39.803+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:40.275 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:39.803+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:40.275 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:39.803+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:40.275 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:39.804+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:40.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:00:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:00:40.274+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 8917 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:00:40.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:40.479+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:40.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:40.479+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:40.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:40.479+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:40.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:40.480+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:41.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:41.023+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:41.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:41.023+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:41.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:41.023+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:41.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:41.025+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:41.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:40.828+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:41.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:40.828+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:41.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:40.828+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:41.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:40.828+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:41.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:41.471+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:41.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:41.471+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:41.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:41.471+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:41.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:41.472+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:42.069 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:41.799+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:42.070 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:41.799+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:42.070 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:41.799+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:42.070 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:41.800+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:42.508 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:42.070+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:42.508 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:42.070+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:42.508 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:42.070+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:42.508 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:42.071+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:42.781 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:42.509+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:42.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:42.509+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:42.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:42.509+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:42.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:42.509+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:43.081 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:42.803+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:43.081 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:42.803+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:43.081 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:42.803+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:43.081 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:42.804+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:43.473 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:43.081+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:43.473 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:43.081+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:43.473 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:43.081+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:43.473 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:43.083+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:43.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:43.496+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:43.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:43.496+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:43.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:43.496+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:43.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:43.497+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:44.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:44.071+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:44.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:44.071+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:44.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:44.071+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:44.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:44.072+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:44.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:43.822+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:44.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:43.822+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:44.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:43.822+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:44.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:43.823+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:44.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:44.486+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:44.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:44.486+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:44.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:44.486+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:44.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:44.487+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:45.275 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:45.039+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:45.275 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:45.039+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:45.275 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:45.039+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:45.275 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:45.040+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:45.275 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:44.818+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:45.275 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:44.818+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:45.276 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:44.818+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:45.276 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:44.818+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:45.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:00:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:00:45.275+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 9011 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:00:45.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:45.458+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:45.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:45.458+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:45.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:45.458+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:45.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:45.459+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:46.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:46.003+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:46.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:46.003+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:46.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:46.003+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:46.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:46.004+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:46.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:45.830+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:46.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:45.830+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:46.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:45.830+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:46.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:45.831+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:46.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:46.488+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:46.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:46.488+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:46.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:46.488+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:46.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:46.489+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:47.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:47.006+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:47.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:47.006+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:47.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:47.006+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:47.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:47.006+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:47.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:46.786+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:47.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:46.786+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:47.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:46.786+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:47.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:46.787+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:47.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:47.468+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:47.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:47.468+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:47.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:47.468+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:47.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:47.469+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:48.080 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:47.965+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:48.080 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:47.965+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:48.080 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:47.965+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:48.080 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:47.966+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:48.081 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:47.822+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:48.081 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:47.822+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:48.081 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:47.822+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:48.081 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:47.823+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:48.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:48.477+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:48.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:48.477+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:48.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:48.477+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:48.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:48.477+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:49.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:48.958+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:49.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:48.958+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:49.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:48.958+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:49.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:48.959+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:49.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:48.824+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:49.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:48.824+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:49.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:48.824+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:49.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:48.825+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:49.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:49.428+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:49.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:49.428+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:49.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:49.428+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:49.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:49.429+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:50.276 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:49.800+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:50.276 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:49.800+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:50.276 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:49.800+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:50.277 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:49.801+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:50.277 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:49.967+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:50.277 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:49.967+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:50.277 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:49.967+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:50.277 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:49.967+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:50.531 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:00:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:00:50.277+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 9101 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:00:50.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:50.396+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:50.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:50.396+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:50.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:50.396+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:50.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:50.396+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:51.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:50.845+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:51.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:50.845+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:51.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:50.845+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:51.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:50.845+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:51.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:50.920+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:51.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:50.920+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:51.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:50.920+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:51.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:50.921+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:51.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:51.349+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:51.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:51.349+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:51.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:51.349+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:51.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:51.350+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:52.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:51.877+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:52.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:51.877+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:52.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:51.877+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:52.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:51.878+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:52.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:51.825+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:52.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:51.825+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:52.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:51.825+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:52.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:51.826+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:52.689 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:52.304+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:52.689 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:52.304+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:52.690 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:52.304+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:52.690 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:52.305+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:53.129 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:52.840+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:53.129 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:52.840+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:53.129 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:52.840+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:53.129 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:52.840+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:53.129 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:52.808+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:53.129 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:52.808+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:53.130 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:52.808+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:53.130 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:52.808+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:53.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:53.287+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:53.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:53.287+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:53.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:53.287+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:53.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:53.288+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:54.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:53.877+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:54.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:53.877+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:54.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:53.877+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:54.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:53.878+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:54.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:53.833+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:54.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:53.833+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:54.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:53.833+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:54.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:53.833+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:54.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:54.335+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:54.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:54.335+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:54.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:54.335+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:54.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:54.336+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:55.277 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:54.838+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:55.278 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:54.838+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:55.278 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:54.838+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:55.278 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:54.839+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:55.278 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:54.856+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:55.278 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:54.856+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:55.278 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:54.856+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:55.278 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:54.857+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:55.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:00:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:00:55.277+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 9191 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:00:55.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:55.324+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:55.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:55.324+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:55.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:55.324+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:55.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:55.325+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:56.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:55.851+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:56.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:55.851+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:56.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:55.851+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:56.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:55.851+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:56.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:55.884+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:56.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:55.885+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:56.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:55.885+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:56.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:55.886+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:56.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:56.368+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:56.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:56.368+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:56.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:56.368+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:56.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:56.368+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:57.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:56.836+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:57.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:56.836+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:57.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:56.837+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:57.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:56.837+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:57.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:56.847+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:57.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:56.847+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:57.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:56.847+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:57.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:56.848+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:57.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:57.377+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:57.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:57.377+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:57.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:57.377+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:57.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:57.377+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:58.174 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:57.877+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:58.174 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:57.877+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:58.175 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:57.877+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:58.175 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:57.878+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:58.175 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:57.835+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:58.175 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:57.835+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:58.175 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:57.835+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:58.175 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:57.836+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:58.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:58.395+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:58.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:58.395+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:58.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:58.395+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:58.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:58.395+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:59.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:58.793+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:59.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:58.793+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:59.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:58.793+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:00:59.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:58.795+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:59.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:58.863+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:59.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:58.863+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:59.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:58.863+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:00:59.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:58.863+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:00:59.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:59.378+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:59.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:59.378+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:59.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:59.378+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:00:59.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:00:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:00:59.378+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:00.279 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:59.822+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:00.279 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:59.823+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:00.279 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:59.823+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:00.279 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:00:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:00:59.823+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:00.279 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:59.882+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:00.279 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:59.882+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:00.279 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:59.882+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:00.279 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:00:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:00:59.883+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:00.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:01:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:01:00.278+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 9275 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:01:00.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:00.391+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:00.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:00.391+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:00.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:00.391+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:00.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:00.392+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:01.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:00.884+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:01.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:00.884+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:01.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:00.884+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:01.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:00.885+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:01.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:00.845+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:01.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:00.845+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:01.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:00.845+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:01.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:00.846+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:01.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:01.422+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:01.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:01.422+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:01.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:01.422+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:01.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:01.423+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:02.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:01.921+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:02.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:01.921+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:02.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:01.921+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:02.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:01.922+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:02.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:01.856+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:02.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:01.856+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:02.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:01.856+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:02.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:01.857+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:02.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:02.407+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:02.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:02.407+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:02.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:02.407+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:02.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:02.407+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:03.222 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:02.842+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:03.222 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:02.842+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:03.222 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:02.842+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:03.223 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:02.842+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:03.223 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:02.967+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:03.223 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:02.967+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:03.223 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:02.967+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:03.223 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:02.968+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:03.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:03.432+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:03.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:03.432+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:03.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:03.432+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:03.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:03.432+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:04.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:03.816+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:04.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:03.816+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:04.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:03.816+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:04.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:03.817+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:04.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:04.003+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:04.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:04.003+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:04.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:04.003+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:04.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:04.003+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:04.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:04.444+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:04.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:04.444+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:04.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:04.444+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:04.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:04.444+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:05.280 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:05.046+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:05.280 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:05.046+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:05.280 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:05.046+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:05.280 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:05.047+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:05.281 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:04.827+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:05.281 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:04.827+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:05.281 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:04.827+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:05.281 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:04.828+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:05.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:01:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:01:05.280+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 9363 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:01:05.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:05.489+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:05.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:05.489+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:05.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:05.489+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:05.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:05.490+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:06.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:06.070+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:06.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:06.070+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:06.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:06.070+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:06.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:06.072+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:06.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:05.874+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:06.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:05.874+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:06.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:05.874+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:06.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:05.874+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:06.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:06.530+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:06.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:06.530+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:06.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:06.530+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:06.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:06.530+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:07.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:07.042+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:07.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:07.042+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:07.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:07.042+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:07.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:07.042+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:07.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:06.922+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:07.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:06.922+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:07.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:06.922+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:07.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:06.922+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:07.909 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:07.540+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:07.910 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:07.540+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:07.910 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:07.540+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:07.910 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:07.541+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:08.268 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:08.054+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:08.268 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:08.054+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:08.268 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:08.054+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:08.268 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:08.055+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:08.269 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:07.909+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:08.269 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:07.909+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:08.269 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:07.909+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:08.269 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:07.910+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:08.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:08.511+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:08.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:08.511+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:08.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:08.511+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:08.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:08.512+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:09.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:09.052+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:09.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:09.052+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:09.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:09.052+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:09.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:09.052+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:09.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:08.878+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:09.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:08.878+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:09.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:08.878+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:09.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:08.879+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:09.851 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:09.532+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:09.852 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:09.532+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:09.852 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:09.532+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:09.852 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:09.533+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:10.281 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:10.099+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:10.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:10.099+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:10.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:10.099+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:10.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:10.100+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:10.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:09.851+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:10.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:09.851+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:10.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:09.851+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:10.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:09.852+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:10.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:01:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:01:10.281+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 9449 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:01:10.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:10.520+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:10.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:10.520+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:10.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:10.520+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:10.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:10.521+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:11.121 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:10.820+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:11.121 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:10.820+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:11.121 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:10.820+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:11.121 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:10.821+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:11.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:11.149+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:11.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:11.149+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:11.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:11.149+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:11.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:11.151+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:11.870 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:11.554+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:11.870 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:11.554+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:11.870 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:11.554+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:11.871 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:11.554+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:12.127 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:11.869+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:12.127 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:11.869+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:12.128 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:11.869+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:12.128 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:11.870+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:12.524 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:12.127+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:12.524 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:12.127+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:12.524 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:12.127+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:12.524 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:12.128+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:12.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:12.523+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:12.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:12.523+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:12.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:12.523+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:12.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:12.524+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:13.098 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:12.821+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:13.098 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:12.821+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:13.098 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:12.821+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:13.098 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:12.822+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:13.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:13.098+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:13.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:13.098+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:13.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:13.098+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:13.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:13.099+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:13.849 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:13.565+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:13.849 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:13.565+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:13.849 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:13.565+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:13.849 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:13.566+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:14.192 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:14.095+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:14.192 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:14.095+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:14.193 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:14.095+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:14.193 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:14.097+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:14.193 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:13.848+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:14.193 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:13.848+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:14.193 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:13.848+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:14.193 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:13.849+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:14.861 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:14.542+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:14.861 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:14.542+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:14.861 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:14.542+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:14.861 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:14.543+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:15.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:14.860+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:15.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:14.860+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:15.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:14.860+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:15.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:14.861+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:15.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:15.079+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:15.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:15.079+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:15.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:15.079+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:15.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:15.080+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:15.544 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:01:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:01:15.283+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 9541 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:01:15.887 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:15.544+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:15.887 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:15.544+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:15.887 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:15.544+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:15.887 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:15.545+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:16.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:15.886+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:16.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:15.886+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:16.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:15.886+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:16.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:15.887+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:16.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:16.036+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:16.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:16.036+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:16.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:16.036+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:16.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:16.037+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:16.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:16.521+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:16.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:16.521+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:16.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:16.521+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:16.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:16.521+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:17.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:16.883+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:17.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:16.883+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:17.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:16.883+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:17.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:16.884+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:17.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:17.069+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:17.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:17.069+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:17.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:17.069+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:17.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:17.070+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:17.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:17.524+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:17.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:17.524+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:17.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:17.524+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:17.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:17.525+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:18.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:18.038+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:18.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:18.038+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:18.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:18.038+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:18.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:18.039+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:18.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:17.899+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:18.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:17.899+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:18.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:17.899+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:18.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:17.900+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:18.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:18.551+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:18.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:18.551+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:18.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:18.551+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:18.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:18.552+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:19.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:19.047+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:19.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:19.047+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:19.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:19.047+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:19.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:19.048+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:19.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:18.912+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:19.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:18.912+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:19.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:18.912+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:19.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:18.913+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:19.853 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:19.538+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:19.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:19.538+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:19.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:19.538+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:19.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:19.539+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:20.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:20.038+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:20.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:20.038+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:20.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:20.038+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:20.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:20.039+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:20.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:19.952+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:20.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:19.952+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:20.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:19.952+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:20.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:19.953+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:20.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:01:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:01:20.284+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 9634 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:01:20.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:20.525+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:20.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:20.525+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:20.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:20.525+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:20.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:20.526+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:21.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:21.010+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:21.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:21.010+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:21.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:21.010+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:21.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:21.011+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:21.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:20.981+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:21.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:20.981+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:21.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:20.981+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:21.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:20.981+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:21.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:21.525+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:21.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:21.525+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:21.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:21.525+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:21.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:21.526+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:22.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:21.941+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:22.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:21.941+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:22.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:21.941+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:22.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:21.942+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:22.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:21.992+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:22.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:21.992+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:22.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:21.992+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:22.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:21.993+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:22.952 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:22.538+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:22.953 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:22.538+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:22.953 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:22.538+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:22.953 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:22.539+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:23.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:22.952+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:23.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:22.952+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:23.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:22.952+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:23.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:22.952+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:23.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:22.968+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:23.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:22.968+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:23.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:22.968+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:23.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:22.969+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:23.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:23.563+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:23.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:23.563+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:23.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:23.563+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:23.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:23.564+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:24.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:23.937+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:24.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:23.937+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:24.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:23.937+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:24.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:23.937+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:24.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:23.923+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:24.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:23.923+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:24.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:23.923+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:24.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:23.924+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:24.936 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:24.546+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:24.936 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:24.546+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:24.936 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:24.546+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:24.936 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:24.547+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:25.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:24.935+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:25.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:24.935+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:25.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:24.935+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:25.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:24.936+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:25.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:24.982+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:25.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:24.982+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:25.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:24.982+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:25.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:24.982+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:25.563 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:01:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:01:25.285+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 9732 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:01:25.923 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:25.562+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:25.923 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:25.562+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:25.923 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:25.562+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:25.923 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:25.563+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:26.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:26.012+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:26.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:26.012+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:26.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:26.012+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:26.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:26.013+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:26.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:25.922+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:26.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:25.922+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:26.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:25.922+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:26.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:25.923+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:26.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:26.522+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:26.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:26.522+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:26.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:26.522+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:26.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:26.523+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:27.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:27.035+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:27.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:27.035+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:27.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:27.035+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:27.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:27.036+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:27.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:26.922+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:27.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:26.922+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:27.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:26.922+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:27.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:26.923+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:27.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:27.501+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:27.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:27.501+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:27.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:27.501+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:27.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:27.502+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:28.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:28.058+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:28.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:28.058+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:28.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:28.058+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:28.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:28.059+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:28.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:27.937+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:28.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:27.937+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:28.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:27.937+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:28.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:27.938+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:28.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:28.493+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:28.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:28.493+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:28.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:28.493+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:28.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:28.494+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:29.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:28.912+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:29.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:28.912+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:29.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:28.912+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:29.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:28.913+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:29.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:29.073+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:29.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:29.073+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:29.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:29.073+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:29.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:29.074+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:29.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:29.450+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:29.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:29.450+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:29.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:29.450+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:29.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:29.451+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:30.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:30.056+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:30.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:30.056+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:30.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:30.056+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:30.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:30.057+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:30.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:29.865+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:30.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:29.865+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:30.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:29.865+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:30.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:29.866+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:30.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:01:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:01:30.286+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 9825 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:01:30.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:30.498+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:30.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:30.498+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:30.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:30.498+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:30.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:30.499+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:31.094 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:30.834+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:31.094 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:30.834+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:31.095 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:30.834+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:31.095 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:30.835+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:31.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:31.094+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:31.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:31.094+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:31.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:31.094+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:31.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:31.095+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:31.885 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:31.546+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:31.885 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:31.546+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:31.885 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:31.546+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:31.885 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:31.547+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:32.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:32.125+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:32.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:32.125+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:32.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:32.125+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:32.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:32.126+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:32.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:31.885+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:32.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:31.885+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:32.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:31.885+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:32.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:31.886+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:32.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:32.525+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:32.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:32.525+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:32.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:32.525+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:32.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:32.526+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:33.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:33.107+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:33.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:33.107+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:33.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:33.107+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:33.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:33.108+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:33.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:32.908+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:33.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:32.908+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:33.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:32.908+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:33.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:32.909+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:33.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:33.535+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:33.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:33.535+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:33.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:33.535+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:33.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:33.535+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:34.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:34.088+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:34.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:34.088+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:34.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:34.088+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:34.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:34.089+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:34.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:33.913+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:34.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:33.913+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:34.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:33.913+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:34.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:33.914+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:34.963 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:34.572+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:34.963 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:34.572+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:34.963 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:34.572+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:34.963 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:34.573+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:35.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:34.962+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:35.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:34.962+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:35.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:34.962+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:35.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:34.963+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:35.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:35.046+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:35.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:35.046+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:35.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:35.046+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:35.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:35.047+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:35.581 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:01:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:01:35.288+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 9921 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:01:35.934 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:35.581+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:35.935 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:35.581+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:35.935 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:35.581+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:35.935 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:35.582+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:36.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:35.934+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:36.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:35.934+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:36.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:35.934+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:36.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:35.934+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:36.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:36.000+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:36.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:36.000+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:36.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:36.000+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:36.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:36.001+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:36.936 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:36.547+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:36.936 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:36.547+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:36.936 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:36.547+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:36.936 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:36.548+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:37.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:36.980+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:37.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:36.980+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:37.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:36.980+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:37.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:36.981+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:37.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:36.935+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:37.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:36.935+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:37.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:36.935+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:37.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:36.936+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:37.950 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:37.559+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:37.950 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:37.559+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:37.950 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:37.559+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:37.950 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:37.560+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:38.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:37.949+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:38.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:37.949+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:38.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:37.949+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:38.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:37.950+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:38.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:38.001+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:38.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:38.001+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:38.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:38.001+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:38.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:38.001+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:38.938 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:38.566+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:38.939 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:38.566+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:38.939 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:38.566+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:38.939 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:38.566+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:39.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:38.938+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:39.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:38.938+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:39.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:38.938+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:39.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:38.939+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:39.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:39.029+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:39.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:39.029+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:39.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:39.029+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:39.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:39.030+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:39.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:39.530+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:39.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:39.530+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:39.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:39.530+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:39.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:39.531+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:40.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:39.980+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:40.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:39.980+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:40.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:39.980+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:40.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:39.981+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:40.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:40.012+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:40.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:40.012+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:40.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:40.012+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:40.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:40.013+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:40.575 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:01:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:01:40.289+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 10013 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:01:40.959 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:40.574+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:40.959 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:40.574+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:40.960 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:40.574+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:40.960 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:40.575+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:41.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:41.046+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:41.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:41.046+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:41.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:41.046+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:41.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:41.046+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:41.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:40.959+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:41.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:40.959+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:41.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:40.959+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:41.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:40.960+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:42.006 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:41.545+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:42.009 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:41.545+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:42.009 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:41.545+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:42.009 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:41.546+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:42.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:42.005+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:42.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:42.005+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:42.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:42.005+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:42.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:42.006+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:42.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:42.079+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:42.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:42.079+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:42.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:42.079+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:42.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:42.080+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:43.005 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:42.536+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:43.006 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:42.536+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:43.006 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:42.536+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:43.006 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:42.537+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:43.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:43.034+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:43.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:43.034+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:43.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:43.034+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:43.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:43.035+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:43.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:43.005+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:43.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:43.005+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:43.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:43.005+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:43.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:43.006+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:44.027 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:43.535+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:44.027 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:43.535+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:44.027 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:43.535+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:44.027 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:43.536+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:44.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:44.042+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:44.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:44.042+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:44.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:44.042+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:44.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:44.043+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:44.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:44.026+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:44.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:44.026+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:44.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:44.026+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:44.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:44.027+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:44.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:44.502+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:44.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:44.502+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:44.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:44.502+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:44.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:44.503+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:45.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:45.064+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:45.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:45.064+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:45.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:45.064+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:45.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:45.065+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:45.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:45.055+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:45.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:45.055+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:45.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:45.055+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:45.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:45.056+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:45.543 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:01:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:01:45.290+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 10110 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:01:46.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:45.543+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:46.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:45.543+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:46.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:45.543+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:46.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:45.544+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:46.530 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:46.095+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:46.530 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:46.095+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:46.531 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:46.095+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:46.531 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:46.096+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:46.531 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:46.087+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:46.531 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:46.087+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:46.531 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:46.087+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:46.531 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:46.088+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:46.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:46.530+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:46.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:46.530+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:46.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:46.530+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:46.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:46.530+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:47.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:47.052+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:47.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:47.052+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:47.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:47.052+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:47.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:47.053+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:47.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:47.141+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:47.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:47.141+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:47.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:47.141+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:47.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:47.141+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:47.985 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:47.565+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:47.985 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:47.565+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:47.985 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:47.565+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:47.985 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:47.566+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:48.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:48.150+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:48.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:48.150+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:48.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:48.150+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:48.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:48.151+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:48.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:48.101+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:48.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:48.101+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:48.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:48.101+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:48.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:48.102+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:49.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:48.532+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:49.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:48.532+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:49.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:48.532+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:49.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:48.533+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:49.531 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:49.189+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:49.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:49.189+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:49.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:49.189+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:49.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:49.190+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:49.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:49.062+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:49.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:49.062+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:49.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:49.062+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:49.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:49.062+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:49.853 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:49.551+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:49.853 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:49.551+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:49.853 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:49.551+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:49.853 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:49.551+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:50.212 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:50.082+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:50.212 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:50.082+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:50.212 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:50.082+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:50.212 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:50.082+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:50.514 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:01:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:01:50.291+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 10203 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:01:50.515 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:50.211+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:50.515 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:50.211+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:50.515 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:50.211+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:50.515 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:50.212+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:50.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:50.514+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:50.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:50.514+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:50.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:50.514+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:50.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:50.515+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:51.493 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:51.220+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:51.493 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:51.220+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:51.493 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:51.220+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:51.493 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:51.221+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:51.493 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:51.078+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:51.493 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:51.078+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:51.493 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:51.078+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:51.493 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:51.079+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:51.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:51.492+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:51.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:51.492+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:51.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:51.492+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:51.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:51.493+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:52.468 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:52.259+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:52.468 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:52.259+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:52.468 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:52.259+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:52.468 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:52.259+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:52.468 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:52.112+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:52.468 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:52.112+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:52.468 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:52.112+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:52.469 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:52.113+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:52.781 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:52.467+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:52.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:52.467+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:52.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:52.467+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:52.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:52.468+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:53.472 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:53.260+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:53.473 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:53.260+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:53.473 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:53.260+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:53.473 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:53.261+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:53.473 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:53.109+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:53.473 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:53.109+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:53.473 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:53.109+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:53.474 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:53.109+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:53.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:53.472+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:53.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:53.472+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:53.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:53.472+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:53.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:53.473+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:54.502 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:54.219+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:54.503 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:54.219+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:54.503 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:54.219+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:54.503 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:54.219+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:54.503 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:54.138+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:54.503 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:54.138+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:54.503 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:54.138+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:54.503 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:54.139+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:54.781 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:54.502+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:54.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:54.502+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:54.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:54.502+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:54.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:54.503+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:55.415 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:01:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:01:55.293+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 10299 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:01:55.415 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:55.243+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:55.415 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:55.243+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:55.416 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:55.243+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:55.416 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:55.244+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:55.416 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:55.098+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:55.416 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:55.098+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:55.416 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:55.098+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:55.416 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:55.100+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:55.781 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:55.517+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:55.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:55.517+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:55.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:55.517+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:55.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:55.518+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:56.471 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:56.230+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:56.472 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:56.230+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:56.472 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:56.230+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:56.472 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:56.231+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:56.472 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:56.054+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:56.472 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:56.054+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:56.472 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:56.054+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:56.472 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:56.055+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:56.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:56.471+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:56.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:56.471+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:56.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:56.471+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:56.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:56.472+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:57.443 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:57.060+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:57.443 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:57.060+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:57.443 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:57.060+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:57.443 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:57.060+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:57.444 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:57.246+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:57.444 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:57.246+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:57.444 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:57.246+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:57.444 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:57.246+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:57.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:57.442+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:57.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:57.442+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:57.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:57.442+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:57.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:57.443+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:58.451 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:58.280+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:58.451 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:58.280+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:58.451 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:58.280+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:58.451 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:58.280+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:58.451 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:58.031+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:58.451 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:58.031+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:58.452 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:58.031+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:58.452 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:58.032+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:58.745 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:58.450+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:58.745 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:58.450+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:58.745 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:58.450+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:58.745 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:58.451+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:59.320 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:59.036+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:59.320 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:59.036+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:59.320 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:59.036+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:01:59.320 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:01:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:01:59.037+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:59.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:59.432+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:59.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:59.432+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:59.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:59.432+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:01:59.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:01:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:01:59.432+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:01:59.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:59.319+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:59.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:59.319+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:59.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:59.319+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:01:59.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:01:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:01:59.320+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:00.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:00.085+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:00.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:00.085+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:00.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:00.085+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:00.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:00.085+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:00.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:02:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:02:00.295+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 10392 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:02:00.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:00.358+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:00.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:00.358+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:00.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:00.358+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:00.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:00.359+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:00.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:00.403+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:00.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:00.403+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:00.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:00.403+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:00.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:00.404+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:01.374 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:01.070+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:01.374 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:01.070+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:01.374 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:01.070+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:01.374 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:01.071+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:01.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:01.448+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:01.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:01.448+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:01.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:01.448+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:01.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:01.449+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:01.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:01.374+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:01.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:01.374+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:01.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:01.374+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:01.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:01.375+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:02.331 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:02.048+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:02.332 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:02.048+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:02.332 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:02.048+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:02.332 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:02.049+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:02.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:02.456+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:02.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:02.456+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:02.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:02.456+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:02.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:02.458+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:02.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:02.332+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:02.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:02.332+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:02.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:02.332+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:02.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:02.333+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:03.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:03.013+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:03.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:03.013+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:03.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:03.013+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:03.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:03.013+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:03.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:03.337+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:03.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:03.337+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:03.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:03.337+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:03.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:03.338+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:03.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:03.469+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:03.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:03.469+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:03.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:03.469+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:03.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:03.470+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:04.312 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:04.055+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:04.312 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:04.055+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:04.312 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:04.055+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:04.312 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:04.056+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:04.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:04.511+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:04.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:04.511+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:04.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:04.511+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:04.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:04.512+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:04.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:04.312+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:04.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:04.312+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:04.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:04.312+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:04.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:04.313+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:05.418 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:02:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:02:05.296+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 10490 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:02:05.419 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:05.101+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:05.419 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:05.101+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:05.419 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:05.101+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:05.419 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:05.102+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:05.419 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:05.276+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:05.419 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:05.276+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:05.420 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:05.276+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:05.420 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:05.277+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:05.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:05.476+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:05.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:05.476+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:05.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:05.476+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:05.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:05.477+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:06.416 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:06.322+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:06.417 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:06.322+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:06.417 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:06.322+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:06.417 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:06.322+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:06.417 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:06.119+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:06.417 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:06.119+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:06.417 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:06.119+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:06.418 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:06.120+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:06.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:06.476+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:06.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:06.476+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:06.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:06.476+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:06.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:06.477+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:07.471 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:07.355+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:07.471 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:07.355+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:07.471 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:07.355+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:07.472 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:07.356+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:07.472 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:07.163+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:07.472 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:07.163+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:07.472 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:07.163+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:07.472 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:07.164+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:07.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:07.472+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:07.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:07.472+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:07.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:07.472+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:07.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:07.472+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:08.467 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:08.311+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:08.467 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:08.312+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:08.467 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:08.312+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:08.467 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:08.312+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:08.467 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:08.212+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:08.468 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:08.212+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:08.468 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:08.212+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:08.468 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:08.213+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:08.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:08.467+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:08.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:08.467+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:08.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:08.467+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:08.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:08.468+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:09.458 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:09.309+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:09.458 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:09.309+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:09.458 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:09.309+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:09.458 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:09.310+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:09.459 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:09.181+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:09.459 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:09.181+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:09.459 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:09.181+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:09.459 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:09.181+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:09.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:09.459+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:09.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:09.459+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:09.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:09.459+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:09.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:09.460+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:10.451 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:02:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:02:10.298+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 10582 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:02:10.451 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:10.352+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:10.451 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:10.352+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:10.451 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:10.352+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:10.452 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:10.353+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:10.452 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:10.198+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:10.452 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:10.198+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:10.452 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:10.198+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:10.452 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:10.199+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:10.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:10.452+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:10.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:10.452+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:10.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:10.452+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:10.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:10.453+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:11.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:11.450+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:11.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:11.450+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:11.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:11.450+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:11.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:11.451+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:11.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:11.396+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:11.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:11.396+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:11.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:11.396+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:11.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:11.397+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:11.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:11.245+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:11.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:11.245+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:11.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:11.245+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:11.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:11.246+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:12.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:12.460+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:12.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:12.460+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:12.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:12.460+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:12.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:12.461+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:12.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:12.387+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:12.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:12.388+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:12.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:12.388+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:12.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:12.388+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:12.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:12.258+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:12.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:12.258+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:12.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:12.258+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:12.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:12.259+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:13.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:13.233+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:13.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:13.233+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:13.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:13.233+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:13.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:13.234+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:13.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:13.365+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:13.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:13.365+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:13.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:13.365+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:13.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:13.366+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:13.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:13.425+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:13.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:13.425+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:13.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:13.425+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:13.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:13.426+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:14.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:14.263+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:14.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:14.263+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:14.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:14.263+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:14.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:14.264+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:14.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:14.413+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:14.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:14.413+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:14.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:14.413+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:14.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:14.414+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:14.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:14.372+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:14.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:14.372+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:14.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:14.372+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:14.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:14.373+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:15.632 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:02:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:02:15.298+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 10678 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:02:15.633 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:15.406+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:15.633 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:15.406+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:15.633 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:15.406+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:15.633 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:15.406+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:15.634 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:15.293+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:15.634 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:15.293+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:15.634 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:15.293+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:15.634 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:15.294+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:15.634 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:15.388+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:15.635 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:15.388+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:15.635 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:15.388+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:15.635 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:15.389+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:16.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:16.262+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:16.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:16.262+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:16.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:16.262+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:16.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:16.263+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:16.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:16.372+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:16.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:16.372+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:16.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:16.372+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:16.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:16.373+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:16.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:16.375+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:16.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:16.375+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:16.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:16.375+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:16.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:16.375+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:17.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:17.256+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:17.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:17.256+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:17.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:17.256+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:17.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:17.257+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:17.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:17.415+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:17.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:17.415+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:17.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:17.415+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:17.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:17.416+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:17.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:17.383+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:17.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:17.383+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:17.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:17.383+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:17.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:17.383+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:18.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:18.295+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:18.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:18.295+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:18.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:18.295+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:18.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:18.296+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:18.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:18.418+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:18.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:18.418+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:18.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:18.418+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:18.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:18.419+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:18.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:18.396+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:18.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:18.396+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:18.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:18.396+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:18.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:18.397+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:19.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:19.402+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:19.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:19.402+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:19.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:19.402+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:19.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:19.403+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:19.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:19.401+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:19.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:19.401+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:19.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:19.401+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:19.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:19.402+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:19.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:19.291+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:19.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:19.291+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:19.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:19.291+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:19.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:19.292+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:20.392 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:02:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:02:20.299+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 10770 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:02:20.393 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:20.260+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:20.393 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:20.260+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:20.393 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:20.260+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:20.393 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:20.261+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:20.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:20.392+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:20.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:20.392+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:20.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:20.392+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:20.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:20.393+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:20.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:20.423+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:20.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:20.423+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:20.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:20.423+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:20.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:20.423+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:21.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:21.440+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:21.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:21.440+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:21.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:21.440+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:21.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:21.441+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:21.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:21.248+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:21.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:21.248+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:21.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:21.248+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:21.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:21.248+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:21.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:21.454+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:21.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:21.454+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:21.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:21.454+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:21.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:21.454+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:22.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:22.486+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:22.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:22.486+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:22.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:22.486+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:22.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:22.487+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:22.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:22.239+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:22.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:22.239+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:22.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:22.239+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:22.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:22.240+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:22.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:22.428+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:22.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:22.428+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:22.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:22.428+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:22.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:22.429+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:23.468 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:23.466+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:23.468 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:23.466+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:23.468 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:23.466+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:23.468 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:23.467+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:23.469 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:23.217+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:23.469 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:23.217+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:23.469 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:23.217+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:23.469 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:23.218+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:23.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:23.468+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:23.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:23.468+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:23.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:23.468+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:23.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:23.469+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:24.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:24.196+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:24.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:24.196+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:24.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:24.196+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:24.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:24.197+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:24.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:24.450+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:24.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:24.450+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:24.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:24.450+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:24.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:24.451+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:24.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:24.447+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:24.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:24.447+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:24.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:24.447+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:24.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:24.447+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:25.448 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:02:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:02:25.300+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 10867 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:02:25.448 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:25.179+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:25.448 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:25.179+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:25.448 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:25.179+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:25.448 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:25.180+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:25.448 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:25.410+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:25.449 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:25.410+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:25.449 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:25.410+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:25.449 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:25.411+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:25.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:25.447+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:25.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:25.447+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:25.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:25.447+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:25.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:25.448+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:26.476 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:26.213+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:26.476 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:26.213+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:26.477 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:26.213+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:26.477 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:26.214+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:26.477 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:26.439+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:26.477 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:26.439+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:26.477 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:26.439+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:26.477 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:26.440+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:26.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:26.475+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:26.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:26.475+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:26.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:26.475+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:26.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:26.476+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:27.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:27.454+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:27.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:27.454+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:27.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:27.454+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:27.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:27.455+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:27.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:27.229+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:27.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:27.229+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:27.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:27.229+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:27.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:27.229+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:27.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:27.427+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:27.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:27.427+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:27.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:27.427+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:27.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:27.429+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:28.406+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:28.406+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:28.406+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:28.407+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:28.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:28.230+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:28.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:28.230+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:28.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:28.230+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:28.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:28.231+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:28.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:28.442+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:28.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:28.442+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:28.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:28.442+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:28.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:28.443+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:29.406 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:29.220+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:29.406 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:29.220+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:29.406 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:29.220+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:29.407 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:29.220+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:29.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:29.406+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:29.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:29.406+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:29.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:29.406+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:29.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:29.407+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:29.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:29.428+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:29.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:29.428+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:29.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:29.428+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:29.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:29.429+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:30.415 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:02:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:02:30.302+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 10960 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:02:30.415 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:30.234+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:30.415 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:30.234+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:30.415 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:30.234+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:30.415 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:30.235+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:30.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:30.448+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:30.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:30.448+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:30.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:30.448+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:30.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:30.449+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:30.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:30.414+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:30.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:30.414+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:30.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:30.414+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:30.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:30.415+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:31.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:31.464+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:31.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:31.464+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:31.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:31.464+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:31.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:31.465+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:31.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:31.477+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:31.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:31.477+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:31.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:31.477+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:31.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:31.477+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:31.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:31.262+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:31.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:31.262+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:31.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:31.262+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:31.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:31.263+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:32.504 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:32.243+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:32.504 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:32.243+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:32.504 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:32.243+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:32.505 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:32.245+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:32.505 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:32.457+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:32.505 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:32.457+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:32.505 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:32.457+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:32.505 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:32.457+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:32.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:32.503+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:32.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:32.503+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:32.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:32.503+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:32.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:32.504+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:33.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:33.222+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:33.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:33.222+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:33.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:33.222+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:33.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:33.223+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:33.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:33.449+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:33.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:33.449+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:33.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:33.449+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:33.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:33.449+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:34.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:33.537+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:34.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:33.537+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:34.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:33.537+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:34.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:33.538+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:34.468 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:34.234+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:34.468 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:34.234+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:34.468 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:34.234+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:34.468 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:34.235+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:34.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:34.526+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:34.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:34.526+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:34.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:34.526+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:34.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:34.527+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:34.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:34.467+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:34.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:34.467+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:34.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:34.467+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:34.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:34.468+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:35.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:02:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:02:35.303+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 11056 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:02:35.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:35.225+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:35.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:35.225+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:35.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:35.225+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:35.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:35.225+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:35.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:35.420+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:35.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:35.420+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:35.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:35.420+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:35.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:35.421+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:36.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:35.556+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:36.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:35.556+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:36.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:35.556+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:36.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:35.556+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:36.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:36.263+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:36.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:36.263+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:36.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:36.263+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:36.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:36.264+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:36.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:36.465+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:36.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:36.465+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:36.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:36.465+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:36.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:36.465+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:37.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:36.582+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:37.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:36.582+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:37.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:36.582+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:37.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:36.583+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:37.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:37.267+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:37.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:37.267+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:37.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:37.267+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:37.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:37.268+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:37.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:37.500+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:37.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:37.500+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:37.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:37.500+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:37.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:37.501+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:38.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:37.572+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:38.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:37.572+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:38.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:37.572+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:38.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:37.573+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:38.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:38.511+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:38.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:38.511+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:38.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:38.511+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:38.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:38.512+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:38.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:38.264+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:38.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:38.264+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:38.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:38.264+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:38.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:38.265+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:39.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:38.556+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:39.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:38.556+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:39.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:38.556+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:39.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:38.557+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:39.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:39.272+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:39.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:39.272+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:39.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:39.272+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:39.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:39.273+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:39.853 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:39.598+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:39.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:39.598+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:39.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:39.598+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:39.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:39.599+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:39.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:39.547+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:39.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:39.547+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:39.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:39.547+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:39.855 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:39.547+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:40.429 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:02:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:02:40.305+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 11149 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:02:40.429 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:40.276+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:40.429 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:40.276+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:40.429 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:40.276+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:40.429 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:40.277+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:40.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:40.573+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:40.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:40.573+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:40.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:40.573+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:40.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:40.574+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:40.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:40.614+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:40.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:40.614+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:40.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:40.614+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:40.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:40.615+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:41.577 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:41.319+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:41.577 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:41.319+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:41.577 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:41.319+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:41.577 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:41.320+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:42.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:41.579+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:42.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:41.579+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:42.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:41.579+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:42.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:41.580+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:42.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:41.577+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:42.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:41.577+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:42.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:41.577+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:42.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:41.577+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:42.595 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:42.327+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:42.596 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:42.327+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:42.596 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:42.327+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:42.596 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:42.328+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:43.013 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:42.608+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:43.014 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:42.608+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:43.014 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:42.608+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:43.014 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:42.609+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:43.014 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:42.595+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:43.014 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:42.595+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:43.014 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:42.595+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:43.014 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:42.596+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:43.550 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:43.287+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:43.550 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:43.287+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:43.550 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:43.287+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:43.550 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:43.288+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:44.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:43.549+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:44.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:43.549+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:44.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:43.550+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:44.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:43.551+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:44.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:43.611+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:44.035 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:43.611+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:44.035 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:43.611+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:44.035 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:43.612+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:44.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:44.266+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:44.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:44.266+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:44.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:44.266+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:44.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:44.267+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:45.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:44.576+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:45.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:44.576+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:45.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:44.576+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:45.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:44.577+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:45.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:44.576+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:45.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:44.576+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:45.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:44.576+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:45.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:44.577+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:45.613 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:02:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:02:45.306+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 11247 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:02:45.613 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:45.307+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:45.613 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:45.307+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:45.613 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:45.307+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:45.613 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:45.308+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:45.613 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:45.546+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:45.613 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:45.546+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:45.613 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:45.546+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:45.614 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:45.547+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:46.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:45.612+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:46.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:45.612+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:46.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:45.612+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:46.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:45.613+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:46.576 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:46.572+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:46.576 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:46.572+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:46.576 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:46.572+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:46.576 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:46.573+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:46.576 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:46.325+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:46.576 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:46.325+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:46.576 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:46.325+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:46.577 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:46.326+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:47.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:46.575+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:47.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:46.575+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:47.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:46.575+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:47.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:46.577+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:47.600 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:47.307+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:47.600 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:47.307+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:47.600 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:47.307+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:47.600 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:47.307+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:47.601 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:47.541+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:47.601 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:47.541+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:47.601 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:47.541+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:47.601 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:47.541+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:48.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:47.600+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:48.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:47.600+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:48.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:47.600+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:48.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:47.601+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:48.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:48.278+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:48.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:48.278+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:48.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:48.278+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:48.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:48.279+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:49.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:48.552+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:49.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:48.552+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:49.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:48.552+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:49.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:48.553+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:49.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:48.534+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:49.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:48.534+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:49.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:48.534+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:49.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:48.535+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:49.518 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:49.307+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:49.518 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:49.307+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:49.518 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:49.307+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:49.518 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:49.308+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:49.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:49.567+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:49.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:49.567+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:49.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:49.567+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:49.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:49.568+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:49.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:49.517+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:49.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:49.517+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:49.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:49.517+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:49.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:49.519+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:50.434 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:02:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:02:50.307+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 11340 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:02:50.435 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:50.350+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:50.435 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:50.350+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:50.435 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:50.350+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:50.435 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:50.350+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:50.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:50.512+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:50.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:50.512+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:50.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:50.512+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:50.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:50.513+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:50.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:50.552+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:50.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:50.552+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:50.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:50.552+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:50.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:50.553+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:51.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:51.547+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:51.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:51.547+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:51.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:51.547+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:51.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:51.548+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:51.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:51.546+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:51.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:51.546+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:51.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:51.546+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:51.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:51.547+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:51.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:51.348+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:51.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:51.348+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:51.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:51.348+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:51.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:51.349+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:52.588 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:52.334+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:52.588 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:52.334+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:52.588 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:52.334+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:52.588 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:52.335+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:53.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:52.587+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:53.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:52.587+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:53.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:52.587+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:53.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:52.587+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:53.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:52.588+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:53.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:52.588+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:53.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:52.588+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:53.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:52.589+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:53.591 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:53.340+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:53.591 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:53.340+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:53.591 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:53.340+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:53.591 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:53.341+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:53.592 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:53.589+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:53.592 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:53.590+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:53.592 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:53.590+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:53.592 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:53.591+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:54.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:53.604+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:54.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:53.604+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:54.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:53.604+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:54.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:53.605+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:54.619 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:54.312+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:54.619 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:54.312+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:54.620 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:54.312+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:54.620 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:54.313+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:55.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:54.629+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:55.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:54.629+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:55.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:54.629+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:55.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:54.629+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:55.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:54.619+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:55.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:54.619+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:55.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:54.619+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:55.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:54.620+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:55.568 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:02:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:02:55.309+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 11436 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:02:55.568 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:55.362+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:55.568 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:55.362+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:55.569 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:55.362+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:55.569 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:55.363+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:56.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:55.585+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:56.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:55.585+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:56.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:55.585+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:56.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:55.586+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:56.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:55.646+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:56.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:55.646+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:56.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:55.646+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:56.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:55.646+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:56.623 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:56.587+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:56.623 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:56.587+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:56.623 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:56.587+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:56.623 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:56.588+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:56.623 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:56.338+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:56.623 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:56.338+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:56.624 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:56.338+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:56.624 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:56.339+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:57.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:56.622+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:57.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:56.622+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:57.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:56.622+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:57.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:56.623+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:57.594 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:57.329+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:57.595 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:57.329+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:57.595 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:57.329+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:57.595 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:57.330+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:58.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:57.620+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:58.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:57.620+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:58.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:57.620+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:58.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:57.620+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:58.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:57.594+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:58.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:57.594+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:58.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:57.594+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:58.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:57.595+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:58.641 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:58.310+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:58.642 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:58.310+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:58.642 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:58.310+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:58.642 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:58.311+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:59.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:58.648+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:59.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:58.648+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:59.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:58.648+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:59.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:58.649+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:59.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:58.658+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:59.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:58.658+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:59.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:58.658+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:59.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:58.659+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:59.600 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:59.335+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:59.601 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:59.335+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:59.601 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:59.335+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:02:59.601 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:02:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:02:59.336+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:59.853 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:59.619+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:59.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:59.619+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:59.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:59.619+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:02:59.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:02:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:02:59.620+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:02:59.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:59.600+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:59.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:59.600+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:59.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:59.600+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:02:59.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:02:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:02:59.601+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:00.435 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:03:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:03:00.310+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 11529 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:03:00.436 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:00.289+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:00.436 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:00.289+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:00.436 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:00.289+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:00.436 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:00.290+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:00.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:00.597+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:00.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:00.597+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:00.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:00.597+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:00.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:00.598+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:00.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:00.589+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:00.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:00.589+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:00.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:00.589+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:00.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:00.589+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:01.559 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:01.290+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:01.560 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:01.290+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:01.560 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:01.290+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:01.560 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:01.291+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:02.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:01.596+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:02.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:01.596+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:02.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:01.596+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:02.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:01.597+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:02.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:01.559+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:02.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:01.559+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:02.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:01.559+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:02.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:01.560+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:02.618 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:02.331+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:02.618 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:02.331+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:02.619 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:02.331+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:02.619 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:02.332+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:02.619 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:02.550+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:02.619 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:02.550+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:02.619 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:02.550+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:02.619 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:02.550+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:03.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:02.618+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:03.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:02.618+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:03.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:02.618+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:03.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:02.619+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:03.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:03.590+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:03.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:03.590+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:03.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:03.590+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:03.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:03.590+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:03.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:03.347+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:03.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:03.347+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:03.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:03.347+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:03.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:03.348+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:03.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:03.571+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:03.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:03.571+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:03.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:03.571+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:03.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:03.572+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:04.579 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:04.305+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:04.580 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:04.305+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:04.580 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:04.305+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:04.580 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:04.306+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:05.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:04.619+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:05.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:04.619+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:05.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:04.619+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:05.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:04.620+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:05.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:04.579+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:05.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:04.579+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:05.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:04.579+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:05.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:04.580+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:05.592 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:03:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:03:05.311+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 11626 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:03:05.593 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:05.353+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:05.593 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:05.353+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:05.593 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:05.353+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:05.593 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:05.353+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:06.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:05.614+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:06.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:05.614+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:06.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:05.614+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:06.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:05.615+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:06.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:05.592+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:06.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:05.592+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:06.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:05.592+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:06.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:05.593+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:06.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:06.367+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:06.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:06.367+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:06.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:06.367+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:06.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:06.367+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:06.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:06.611+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:06.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:06.611+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:06.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:06.611+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:06.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:06.612+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:06.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:06.581+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:06.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:06.581+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:06.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:06.581+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:06.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:06.582+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:07.622 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:07.320+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:07.622 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:07.320+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:07.622 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:07.320+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:07.622 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:07.321+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:08.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:07.647+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:08.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:07.647+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:08.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:07.647+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:08.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:07.647+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:08.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:07.621+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:08.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:07.621+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:08.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:07.621+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:08.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:07.622+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:08.670 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:08.609+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:08.671 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:08.609+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:08.671 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:08.609+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:08.671 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:08.610+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:08.671 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:08.360+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:08.671 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:08.360+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:08.671 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:08.360+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:08.671 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:08.361+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:09.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:08.670+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:09.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:08.670+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:09.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:08.670+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:09.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:08.671+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:09.649 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:09.369+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:09.650 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:09.369+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:09.650 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:09.369+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:09.650 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:09.370+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:09.650 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:09.614+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:09.650 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:09.614+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:09.650 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:09.614+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:09.651 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:09.615+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:10.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:09.649+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:10.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:09.649+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:10.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:09.649+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:10.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:09.650+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:10.434 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:03:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:03:10.313+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 11720 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:03:10.435 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:10.355+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:10.435 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:10.355+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:10.435 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:10.355+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:10.435 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:10.355+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:10.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:10.614+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:10.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:10.614+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:10.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:10.614+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:10.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:10.614+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:10.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:10.574+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:10.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:10.574+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:10.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:10.574+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:10.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:10.574+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:11.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:11.562+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:11.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:11.562+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:11.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:11.562+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:11.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:11.563+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:11.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:11.381+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:11.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:11.381+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:11.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:11.381+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:11.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:11.382+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:11.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:11.594+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:11.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:11.594+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:11.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:11.594+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:11.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:11.595+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:12.601 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:12.526+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:12.601 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:12.526+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:12.601 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:12.526+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:12.602 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:12.527+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:12.602 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:12.348+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:12.602 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:12.348+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:12.602 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:12.348+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:12.602 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:12.349+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:13.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:12.600+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:13.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:12.600+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:13.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:12.600+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:13.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:12.601+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:13.637 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:13.566+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:13.638 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:13.566+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:13.638 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:13.566+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:13.638 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:13.567+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:13.638 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:13.331+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:13.638 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:13.331+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:13.638 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:13.331+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:13.638 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:13.332+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:14.000 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:13.636+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:14.000 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:13.636+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:14.000 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:13.636+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:14.000 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:13.637+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:14.589 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:14.283+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:14.589 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:14.283+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:14.589 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:14.283+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:14.590 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:14.283+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:15.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:14.588+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:15.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:14.588+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:15.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:14.588+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:15.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:14.589+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:15.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:14.604+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:15.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:14.604+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:15.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:14.604+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:15.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:14.605+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:15.630 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:03:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:03:15.314+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 11815 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:03:15.630 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:15.295+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:15.630 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:15.295+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:15.630 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:15.295+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:15.631 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:15.296+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:16.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:15.629+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:16.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:15.629+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:16.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:15.629+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:16.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:15.631+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:16.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:15.632+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:16.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:15.632+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:16.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:15.632+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:16.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:15.633+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:16.592 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:16.297+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:16.593 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:16.297+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:16.593 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:16.297+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:16.593 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:16.298+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:17.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:16.592+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:17.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:16.592+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:17.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:16.592+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:17.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:16.593+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:17.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:16.600+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:17.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:16.601+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:17.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:16.601+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:17.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:16.601+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:17.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:17.258+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:17.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:17.258+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:17.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:17.258+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:17.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:17.259+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:18.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:17.611+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:18.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:17.611+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:18.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:17.611+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:18.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:17.611+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:18.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:17.643+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:18.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:17.643+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:18.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:17.643+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:18.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:17.644+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:18.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:18.241+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:18.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:18.241+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:18.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:18.241+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:18.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:18.241+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:19.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:18.567+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:19.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:18.567+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:19.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:18.567+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:19.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:18.568+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:19.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:18.658+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:19.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:18.658+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:19.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:18.658+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:19.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:18.659+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:19.527 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:19.238+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:19.527 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:19.238+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:19.527 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:19.238+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:19.528 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:19.239+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:19.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:19.559+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:19.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:19.559+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:19.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:19.559+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:19.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:19.559+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:19.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:19.656+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:19.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:19.656+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:19.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:19.656+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:19.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:19.657+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:20.440 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:03:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:03:20.316+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 11909 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:03:20.440 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:20.231+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:20.440 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:20.231+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:20.440 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:20.231+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:20.440 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:20.231+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:20.781 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:20.549+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:20.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:20.549+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:20.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:20.549+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:20.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:20.549+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:20.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:20.687+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:20.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:20.687+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:20.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:20.687+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:20.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:20.687+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:21.519 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:21.220+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:21.519 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:21.220+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:21.519 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:21.220+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:21.519 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:21.221+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:21.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:21.674+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:21.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:21.674+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:21.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:21.674+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:21.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:21.676+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:21.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:21.519+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:21.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:21.519+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:21.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:21.519+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:21.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:21.519+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:22.479 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:22.207+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:22.479 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:22.207+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:22.479 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:22.207+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:22.479 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:22.208+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:22.774 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:22.479+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:22.774 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:22.479+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:22.774 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:22.479+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:22.774 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:22.480+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:22.775 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:22.715+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:22.775 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:22.715+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:22.775 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:22.715+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:22.775 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:22.716+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:23.499 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:23.197+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:23.499 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:23.197+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:23.499 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:23.197+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:23.499 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:23.198+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:23.758 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:23.526+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:23.759 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:23.526+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:23.759 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:23.526+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:23.759 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:23.526+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:24.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:23.759+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:24.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:23.759+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:24.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:23.759+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:24.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:23.760+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:24.488 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:24.185+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:24.488 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:24.185+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:24.488 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:24.185+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:24.488 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:24.186+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:24.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:24.488+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:24.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:24.488+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:24.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:24.488+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:24.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:24.489+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:25.222 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:24.790+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:25.222 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:24.790+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:25.222 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:24.790+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:25.223 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:24.791+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:25.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:03:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:03:25.318+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 12006 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:03:25.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:25.223+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:25.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:25.223+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:25.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:25.223+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:25.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:25.224+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:25.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:25.451+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:25.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:25.451+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:25.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:25.451+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:25.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:25.452+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:26.237 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:25.803+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:26.237 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:25.803+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:26.237 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:25.803+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:26.237 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:25.804+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:26.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:26.238+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:26.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:26.238+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:26.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:26.238+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:26.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:26.238+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:26.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:26.433+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:26.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:26.433+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:26.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:26.433+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:26.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:26.433+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:27.206 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:26.841+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:27.206 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:26.841+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:27.207 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:26.841+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:27.207 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:26.842+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:27.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:27.449+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:27.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:27.449+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:27.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:27.449+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:27.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:27.449+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:27.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:27.207+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:27.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:27.207+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:27.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:27.207+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:27.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:27.208+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:28.162 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:27.855+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:28.162 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:27.855+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:28.162 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:27.855+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:28.162 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:27.856+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:28.413 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:28.162+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:28.413 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:28.162+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:28.413 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:28.162+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:28.413 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:28.163+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:28.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:28.413+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:28.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:28.413+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:28.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:28.413+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:28.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:28.414+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:29.206 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:28.884+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:29.206 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:28.884+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:29.206 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:28.884+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:29.206 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:28.885+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:29.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:29.207+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:29.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:29.207+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:29.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:29.207+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:29.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:29.208+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:29.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:29.410+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:29.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:29.410+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:29.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:29.410+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:29.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:29.411+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:30.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:29.844+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:30.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:29.844+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:30.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:29.844+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:30.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:29.844+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:30.419 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:03:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:03:30.319+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 12100 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:03:30.419 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:30.239+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:30.419 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:30.239+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:30.419 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:30.239+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:30.419 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:30.240+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:30.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:30.419+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:30.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:30.419+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:30.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:30.419+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:30.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:30.421+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:31.248 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:30.882+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:31.248 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:30.882+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:31.249 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:30.882+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:31.249 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:30.883+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:31.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:31.249+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:31.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:31.249+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:31.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:31.249+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:31.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:31.250+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:31.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:31.470+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:31.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:31.470+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:31.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:31.470+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:31.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:31.470+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:32.247 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:31.927+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:32.247 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:31.927+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:32.248 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:31.927+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:32.248 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:31.927+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:32.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:32.478+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:32.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:32.478+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:32.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:32.478+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:32.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:32.479+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:32.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:32.248+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:32.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:32.248+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:32.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:32.248+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:32.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:32.248+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:33.269 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:32.914+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:33.269 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:32.914+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:33.270 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:32.914+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:33.270 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:32.915+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:33.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:33.270+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:33.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:33.270+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:33.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:33.270+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:33.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:33.270+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:33.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:33.500+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:33.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:33.500+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:33.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:33.500+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:33.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:33.501+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:34.182 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:33.922+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:34.182 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:33.922+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:34.182 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:33.922+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:34.182 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:33.923+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:34.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:34.264+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:34.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:34.264+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:34.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:34.264+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:34.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:34.265+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:34.967 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:34.544+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:34.967 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:34.544+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:34.967 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:34.544+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:34.967 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:34.544+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:35.279 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:34.966+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:35.279 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:34.966+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:35.279 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:34.966+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:35.279 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:34.967+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:35.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:03:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:03:35.320+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 12194 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:03:35.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:35.503+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:35.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:35.503+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:35.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:35.503+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:35.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:35.504+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:35.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:35.279+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:35.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:35.279+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:35.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:35.279+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:35.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:35.279+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:36.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:35.977+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:36.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:35.977+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:36.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:35.977+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:36.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:35.979+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:36.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:36.469+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:36.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:36.469+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:36.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:36.469+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:36.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:36.470+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:36.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:36.306+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:36.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:36.306+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:36.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:36.306+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:36.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:36.307+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:37.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:36.972+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:37.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:36.972+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:37.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:36.972+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:37.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:36.973+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:37.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:37.285+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:37.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:37.285+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:37.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:37.285+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:37.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:37.286+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:37.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:37.464+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:37.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:37.464+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:37.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:37.464+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:37.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:37.465+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:38.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:37.971+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:38.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:37.971+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:38.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:37.971+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:38.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:37.972+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:38.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:38.454+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:38.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:38.454+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:38.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:38.454+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:38.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:38.455+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:38.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:38.330+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:38.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:38.330+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:38.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:38.330+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:38.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:38.331+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:39.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:38.961+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:39.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:38.961+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:39.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:38.961+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:39.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:38.962+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:39.696 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:39.356+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:39.696 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:39.356+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:39.696 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:39.356+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:39.696 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:39.357+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:39.696 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:39.477+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:39.697 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:39.477+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:39.697 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:39.477+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:39.697 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:39.478+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:40.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:39.941+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:40.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:39.941+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:40.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:39.941+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:40.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:39.941+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:40.445 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:03:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:03:40.321+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 12288 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:03:40.446 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:40.344+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:40.446 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:40.345+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:40.446 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:40.345+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:40.446 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:40.345+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:40.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:40.456+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:40.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:40.456+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:40.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:40.456+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:40.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:40.457+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:41.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:40.904+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:41.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:40.904+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:41.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:40.904+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:41.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:40.905+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:41.648 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:41.367+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:41.649 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:41.367+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:41.649 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:41.367+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:41.649 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:41.368+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:41.649 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:41.470+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:41.649 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:41.470+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:41.649 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:41.470+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:41.650 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:41.471+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:42.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:41.920+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:42.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:41.920+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:42.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:41.920+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:42.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:41.921+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:42.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:42.397+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:42.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:42.397+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:42.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:42.397+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:42.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:42.398+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:42.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:42.491+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:42.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:42.491+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:42.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:42.491+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:42.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:42.492+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:43.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:42.900+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:43.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:42.900+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:43.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:42.900+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:43.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:42.900+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:43.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:43.385+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:43.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:43.385+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:43.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:43.385+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:43.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:43.386+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:43.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:43.446+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:43.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:43.446+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:43.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:43.446+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:43.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:43.447+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:44.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:43.909+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:44.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:43.909+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:44.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:43.909+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:44.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:43.909+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:44.720 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:44.369+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:44.720 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:44.369+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:44.721 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:44.369+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:44.721 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:44.370+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:44.721 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:44.430+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:44.721 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:44.430+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:44.721 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:44.430+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:44.721 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:44.430+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:45.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:44.928+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:45.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:44.928+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:45.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:44.928+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:45.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:44.929+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:45.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:03:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:03:45.323+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 12385 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:03:45.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:45.413+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:45.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:45.413+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:45.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:45.413+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:45.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:45.414+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:45.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:45.335+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:45.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:45.335+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:45.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:45.335+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:45.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:45.336+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:46.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:45.932+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:46.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:45.932+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:46.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:45.932+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:46.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:45.933+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:46.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:46.394+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:46.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:46.394+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:46.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:46.394+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:46.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:46.395+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:46.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:46.315+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:46.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:46.315+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:46.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:46.315+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:46.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:46.316+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:47.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:46.950+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:47.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:46.950+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:47.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:46.950+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:47.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:46.951+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:47.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:47.392+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:47.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:47.392+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:47.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:47.392+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:47.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:47.392+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:47.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:47.361+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:47.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:47.361+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:47.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:47.361+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:47.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:47.362+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:48.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:47.955+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:48.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:47.955+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:48.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:47.955+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:48.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:47.956+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:48.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:48.416+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:48.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:48.416+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:48.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:48.416+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:48.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:48.417+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:48.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:48.345+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:48.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:48.345+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:48.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:48.345+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:48.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:48.346+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:49.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:48.956+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:49.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:48.956+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:49.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:48.956+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:49.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:48.957+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:49.777 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:49.463+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:49.777 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:49.463+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:49.777 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:49.463+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:49.777 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:49.464+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:49.778 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:49.377+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:49.778 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:49.377+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:49.778 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:49.377+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:49.778 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:49.378+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:50.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:49.911+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:50.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:49.911+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:50.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:49.911+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:50.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:49.911+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:50.448 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:03:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:03:50.324+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 12478 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:03:50.448 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:50.338+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:50.448 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:50.338+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:50.448 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:50.338+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:50.448 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:50.339+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:50.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:50.502+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:50.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:50.502+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:50.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:50.502+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:50.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:50.503+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:51.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:50.947+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:51.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:50.947+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:51.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:50.947+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:51.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:50.947+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:51.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:51.368+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:51.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:51.368+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:51.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:51.368+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:51.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:51.369+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:51.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:51.537+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:51.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:51.537+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:51.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:51.537+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:51.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:51.538+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:52.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:51.985+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:52.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:51.985+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:52.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:51.985+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:52.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:51.986+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:52.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:52.337+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:52.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:52.337+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:52.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:52.337+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:52.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:52.338+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:52.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:52.545+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:52.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:52.545+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:52.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:52.545+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:52.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:52.546+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:53.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:52.957+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:53.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:52.957+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:53.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:52.957+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:53.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:52.957+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:53.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:53.379+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:53.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:53.379+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:53.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:53.379+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:53.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:53.380+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:53.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:53.546+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:53.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:53.546+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:53.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:53.546+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:53.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:53.547+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:54.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:53.937+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:54.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:53.937+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:54.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:53.937+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:54.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:53.938+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:54.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:54.345+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:54.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:54.345+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:54.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:54.345+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:54.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:54.346+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:54.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:54.523+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:54.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:54.523+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:54.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:54.523+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:54.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:54.523+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:55.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:54.941+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:55.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:54.941+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:55.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:54.941+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:55.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:54.942+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:55.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:03:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:03:55.326+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 12572 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:03:55.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:55.534+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:55.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:55.534+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:55.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:55.534+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:55.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:55.535+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:55.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:55.347+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:55.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:55.347+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:55.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:55.347+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:55.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:55.348+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:56.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:55.926+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:56.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:55.926+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:56.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:55.926+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:56.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:55.927+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:56.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:56.381+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:56.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:56.381+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:56.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:56.381+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:56.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:56.382+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:56.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:56.510+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:56.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:56.510+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:56.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:56.510+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:56.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:56.510+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:57.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:56.966+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:57.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:56.966+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:57.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:56.966+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:57.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:56.967+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:57.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:57.517+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:57.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:57.517+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:57.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:57.517+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:57.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:57.518+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:57.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:57.421+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:57.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:57.421+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:57.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:57.421+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:57.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:57.422+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:58.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:57.971+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:58.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:57.971+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:58.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:57.971+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:58.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:57.972+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:58.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:58.559+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:58.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:58.559+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:58.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:58.559+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:58.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:58.560+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:58.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:58.400+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:58.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:58.400+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:58.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:58.400+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:58.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:58.401+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:59.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:58.932+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:59.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:58.932+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:59.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:58.932+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:03:59.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:58.933+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:59.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:59.536+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:59.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:59.536+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:59.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:59.536+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:03:59.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:03:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:03:59.537+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:03:59.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:59.444+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:59.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:59.444+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:59.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:59.444+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:03:59.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:03:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:03:59.445+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:00.147 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:59.963+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:00.147 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:59.963+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:00.147 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:59.963+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:00.147 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:03:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:03:59.963+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:00.449 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:04:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:04:00.327+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 12665 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:04:00.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:00.493+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:00.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:00.493+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:00.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:00.493+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:00.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:00.494+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:00.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:00.494+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:00.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:00.494+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:00.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:00.494+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:00.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:00.495+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:01.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:00.953+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:01.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:00.953+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:01.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:00.953+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:01.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:00.954+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:01.532 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-24T10:04:01.530+0000 7fcba9eb1700 0 monclient(hunting): authenticate timed out after 300 2024-03-24T10:04:01.532 INFO:teuthology.orchestra.run.smithi022.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-03-24T10:04:01.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:01.468+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:01.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:01.468+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:01.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:01.468+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:01.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:01.469+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:01.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:01.504+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:01.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:01.504+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:01.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:01.504+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:01.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:01.505+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:02.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:01.932+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:02.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:01.932+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:02.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:01.932+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:02.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:01.932+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:02.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:04:02.475 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephadm.py", line 1364, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephadm.py", line 39, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi022 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -e sha1=21b3bd672163092b4e542f772797fcba02d3d4de -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2024-03-24T10:04:02.686 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=3ece1145024045678ceb6d1e7ef7e150 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephadm.py", line 1364, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephadm.py", line 39, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi022 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -e sha1=21b3bd672163092b4e542f772797fcba02d3d4de -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2024-03-24T10:04:02.689 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-24T10:04:02.700 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-24T10:04:02.700 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T10:04:02.700 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:04:02.722 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T10:04:02.722 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:04:02.777 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:04:02.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:02.510+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:02.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:02.510+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:02.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:02.510+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:02.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:02.510+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:02.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:02.532+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:02.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:02.532+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:02.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:02.532+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:02.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:02.533+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:03.152 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:02.892+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:03.153 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:02.892+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:03.153 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:02.892+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:03.153 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:02.893+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:03.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:03.507+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:03.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:03.507+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:03.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:03.507+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:03.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:03.508+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:03.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:03.517+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:03.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:03.517+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:03.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:03.517+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:03.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:03.517+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:04.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:03.894+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:04.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:03.894+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:04.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:03.894+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:04.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:03.895+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:04.901 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:04.536+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:04.901 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:04.536+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:04.901 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:04.536+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:04.901 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:04.537+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:04.901 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:04.558+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:04.902 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:04.558+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:04.902 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:04.558+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:04.902 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:04.559+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:05.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:04.922+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:05.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:04.922+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:05.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:04.922+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:05.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:04.923+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:05.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:04:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:04:05.328+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 12762 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:04:05.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:05.524+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:05.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:05.524+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:05.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:05.524+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:05.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:05.525+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:05.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:05.574+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:05.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:05.574+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:05.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:05.574+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:05.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:05.575+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:06.224 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:05.884+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:06.225 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:05.884+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:06.225 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:05.884+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:06.225 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:05.884+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:06.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:06.601+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:06.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:06.601+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:06.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:06.601+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:06.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:06.602+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:06.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:06.500+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:06.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:06.500+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:06.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:06.500+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:06.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:06.501+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:07.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:06.863+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:07.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:06.863+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:07.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:06.863+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:07.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:06.863+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:07.882 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:07.549+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:07.882 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:07.549+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:07.883 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:07.549+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:07.883 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:07.550+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:07.883 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:07.608+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:07.883 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:07.608+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:07.883 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:07.608+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:07.884 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:07.608+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:08.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:07.881+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:08.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:07.881+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:08.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:07.881+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:08.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:07.882+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:08.903 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:08.592+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:08.904 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:08.592+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:08.904 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:08.592+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:08.904 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:08.593+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:08.905 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:08.578+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:08.905 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:08.578+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:08.905 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:08.578+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:08.905 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:08.578+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:09.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:08.903+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:09.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:08.903+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:09.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:08.903+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:09.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:08.904+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:09.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:09.585+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:09.855 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:09.585+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:09.855 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:09.585+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:09.855 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:09.586+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:09.856 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:09.571+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:09.856 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:09.571+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:09.856 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:09.571+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:09.856 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:09.572+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:10.147 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:09.952+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:10.147 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:09.952+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:10.147 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:09.952+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:10.148 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:09.952+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:10.453 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:04:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:04:10.330+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 12855 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:04:10.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:10.562+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:10.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:10.562+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:10.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:10.562+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:10.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:10.563+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:10.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:10.597+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:10.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:10.597+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:10.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:10.597+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:10.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:10.598+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:11.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:10.959+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:11.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:10.959+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:11.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:10.959+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:11.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:10.960+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:11.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:11.517+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:11.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:11.517+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:11.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:11.517+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:11.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:11.517+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:11.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:11.552+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:11.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:11.552+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:11.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:11.552+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:11.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:11.553+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:12.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:11.996+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:12.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:11.996+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:12.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:11.996+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:12.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:11.996+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:12.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:12.508+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:12.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:12.508+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:12.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:12.508+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:12.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:12.509+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:12.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:12.545+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:12.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:12.545+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:12.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:12.546+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:12.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:12.547+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:13.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:13.020+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:13.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:13.020+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:13.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:13.020+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:13.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:13.021+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:13.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:13.498+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:13.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:13.498+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:13.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:13.498+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:13.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:13.499+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:13.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:13.518+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:13.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:13.518+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:13.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:13.518+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:13.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:13.519+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:14.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:14.030+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:14.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:14.030+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:14.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:14.030+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:14.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:14.031+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:14.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:14.561+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:14.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:14.561+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:14.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:14.561+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:14.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:14.562+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:14.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:14.501+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:14.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:14.501+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:14.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:14.501+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:14.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:14.502+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:15.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:15.043+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:15.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:15.043+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:15.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:15.043+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:15.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:15.044+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:15.604 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:04:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:04:15.331+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 12950 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:04:15.605 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:15.458+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:15.605 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:15.458+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:15.605 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:15.458+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:15.605 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:15.459+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:16.031 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:15.604+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:16.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:15.604+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:16.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:15.604+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:16.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:15.605+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:16.466 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:16.052+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:16.467 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:16.052+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:16.467 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:16.052+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:16.467 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:16.053+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:16.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:16.466+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:16.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:16.466+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:16.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:16.466+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:16.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:16.467+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:16.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:16.588+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:16.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:16.588+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:16.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:16.588+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:16.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:16.589+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:17.438 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:17.046+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:17.438 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:17.046+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:17.439 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:17.046+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:17.439 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:17.047+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:17.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:17.438+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:17.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:17.438+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:17.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:17.438+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:17.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:17.439+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:17.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:17.542+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:17.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:17.542+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:17.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:17.542+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:17.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:17.543+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:18.393 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:18.060+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:18.394 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:18.060+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:18.394 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:18.060+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:18.394 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:18.060+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:18.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:18.393+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:18.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:18.393+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:18.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:18.393+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:18.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:18.394+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:18.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:18.570+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:18.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:18.570+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:18.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:18.570+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:18.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:18.571+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:19.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:19.029+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:19.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:19.029+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:19.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:19.029+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:19.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:19.030+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:19.600 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:19.401+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:19.600 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:19.401+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:19.600 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:19.401+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:19.601 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:19.401+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:19.853 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:19.599+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:19.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:19.599+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:19.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:19.599+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:19.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:19.601+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:20.148 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:20.035+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:20.148 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:20.035+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:20.148 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:20.035+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:20.149 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:20.035+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:20.454 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:04:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:04:20.333+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 13044 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:04:20.455 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:20.359+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:20.455 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:20.359+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:20.455 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:20.359+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:20.456 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:20.360+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:20.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:20.617+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:20.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:20.617+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:20.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:20.617+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:20.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:20.617+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:21.356 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:21.060+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:21.356 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:21.060+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:21.356 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:21.060+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:21.357 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:21.061+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:21.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:21.355+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:21.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:21.355+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:21.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:21.355+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:21.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:21.356+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:21.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:21.574+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:21.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:21.574+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:21.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:21.574+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:21.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:21.575+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:22.374 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:22.094+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:22.375 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:22.094+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:22.375 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:22.094+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:22.375 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:22.095+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:22.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:22.374+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:22.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:22.374+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:22.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:22.374+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:22.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:22.375+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:22.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:22.554+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:22.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:22.554+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:22.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:22.554+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:22.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:22.554+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:23.334 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:23.074+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:23.334 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:23.074+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:23.334 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:23.074+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:23.334 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:23.075+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:23.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:23.333+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:23.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:23.333+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:23.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:23.333+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:23.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:23.334+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:23.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:23.575+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:23.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:23.575+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:23.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:23.575+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:23.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:23.576+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:24.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:24.358+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:24.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:24.358+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:24.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:24.358+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:24.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:24.359+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:24.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:24.114+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:24.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:24.114+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:24.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:24.114+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:24.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:24.115+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:24.977 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:24.590+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:24.977 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:24.590+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:24.978 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:24.590+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:24.978 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:24.591+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:25.335 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:25.320+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:25.335 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:25.320+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:25.335 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:25.320+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:25.336 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:25.320+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:25.336 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:25.121+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:25.336 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:25.121+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:25.336 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:25.121+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:25.337 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:25.122+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:25.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:04:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:04:25.335+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 13141 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:04:25.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:25.571+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:25.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:25.571+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:25.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:25.571+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:25.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:25.572+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:26.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:26.319+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:26.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:26.319+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:26.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:26.319+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:26.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:26.320+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:26.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:26.108+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:26.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:26.108+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:26.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:26.108+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:26.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:26.108+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:27.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:26.552+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:27.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:26.552+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:27.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:26.552+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:27.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:26.553+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:27.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:27.323+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:27.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:27.323+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:27.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:27.323+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:27.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:27.324+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:27.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:27.135+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:27.535 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:27.135+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:27.535 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:27.135+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:27.535 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:27.136+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:28.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:27.602+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:28.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:27.602+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:28.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:27.602+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:28.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:27.603+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:28.328+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:28.328+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:28.328+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:28.329+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:28.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:28.086+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:28.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:28.086+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:28.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:28.086+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:28.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:28.087+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:29.031 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:28.581+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:29.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:28.581+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:29.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:28.581+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:29.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:28.582+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:29.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:29.103+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:29.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:29.103+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:29.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:29.103+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:29.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:29.104+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:29.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:29.283+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:29.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:29.283+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:29.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:29.283+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:29.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:29.283+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:29.853 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:29.598+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:29.853 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:29.598+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:29.853 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:29.598+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:29.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:29.598+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:30.462 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:04:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:04:30.336+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 13235 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:04:30.463 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:30.136+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:30.463 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:30.136+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:30.463 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:30.136+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:30.463 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:30.137+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:30.463 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:30.276+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:30.463 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:30.276+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:30.463 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:30.276+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:30.464 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:30.276+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:30.781 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:30.578+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:30.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:30.578+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:30.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:30.578+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:30.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:30.579+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:31.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:31.234+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:31.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:31.234+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:31.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:31.234+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:31.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:31.234+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:31.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:31.145+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:31.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:31.145+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:31.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:31.145+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:31.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:31.146+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:32.031 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:31.607+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:32.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:31.607+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:32.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:31.607+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:32.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:31.607+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:32.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:32.193+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:32.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:32.193+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:32.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:32.193+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:32.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:32.194+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:32.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:32.134+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:32.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:32.134+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:32.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:32.134+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:32.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:32.135+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:33.031 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:32.611+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:33.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:32.611+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:33.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:32.611+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:33.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:32.611+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:33.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:33.105+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:33.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:33.105+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:33.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:33.105+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:33.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:33.106+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:33.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:33.231+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:33.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:33.231+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:33.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:33.231+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:33.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:33.231+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:33.872 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:33.612+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:33.872 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:33.612+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:33.872 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:33.612+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:33.872 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:33.613+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:34.207 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:34.096+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:34.207 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:34.096+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:34.208 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:34.096+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:34.208 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:34.096+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:34.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:34.207+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:34.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:34.207+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:34.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:34.207+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:34.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:34.208+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:35.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:34.582+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:35.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:34.582+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:35.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:34.582+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:35.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:34.583+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:35.338 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:35.085+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:35.339 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:35.085+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:35.339 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:35.085+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:35.339 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:35.085+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:35.339 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:35.251+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:35.339 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:35.251+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:35.340 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:35.251+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:35.340 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:35.252+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:35.781 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:04:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:04:35.338+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 13330 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:04:35.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:35.549+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:35.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:35.549+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:35.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:35.549+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:35.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:35.550+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:36.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:36.097+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:36.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:36.097+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:36.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:36.097+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:36.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:36.098+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:36.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:36.223+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:36.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:36.223+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:36.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:36.223+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:36.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:36.223+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:36.945 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:36.536+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:36.945 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:36.536+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:36.945 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:36.536+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:36.945 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:36.537+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:37.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:37.064+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:37.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:37.064+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:37.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:37.064+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:37.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:37.066+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:37.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:37.238+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:37.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:37.238+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:37.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:37.238+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:37.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:37.238+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:38.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:37.559+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:38.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:37.559+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:38.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:37.559+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:38.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:37.560+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:38.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:38.086+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:38.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:38.086+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:38.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:38.086+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:38.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:38.087+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:38.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:38.286+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:38.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:38.286+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:38.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:38.286+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:38.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:38.287+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:38.587+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:38.587+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:38.587+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:38.588+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:39.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:39.062+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:39.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:39.062+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:39.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:39.062+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:39.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:39.063+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:39.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:39.276+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:39.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:39.276+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:39.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:39.276+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:39.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:39.277+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:39.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:39.589+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:39.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:39.589+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:39.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:39.589+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:39.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:39.590+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:40.149 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:40.056+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:40.149 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:40.056+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:40.149 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:40.056+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:40.149 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:40.056+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:40.462 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:04:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:04:40.339+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 13424 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:04:40.462 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:40.243+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:40.463 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:40.243+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:40.463 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:40.243+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:40.463 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:40.244+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:40.781 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:40.635+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:40.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:40.635+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:40.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:40.635+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:40.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:40.637+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:41.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:41.207+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:41.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:41.207+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:41.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:41.207+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:41.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:41.207+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:41.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:41.029+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:41.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:41.029+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:41.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:41.029+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:41.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:41.030+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:42.030 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:41.647+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:42.030 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:41.647+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:42.030 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:41.647+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:42.031 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:41.648+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:42.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:42.258+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:42.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:42.258+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:42.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:42.258+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:42.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:42.258+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:42.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:42.031+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:42.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:42.031+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:42.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:42.031+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:42.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:42.031+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:43.022 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:42.635+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:43.022 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:42.635+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:43.023 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:42.635+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:43.023 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:42.636+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:43.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:43.022+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:43.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:43.022+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:43.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:43.022+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:43.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:43.023+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:43.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:43.240+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:43.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:43.240+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:43.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:43.240+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:43.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:43.241+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:44.010 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:43.673+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:44.010 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:43.673+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:44.010 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:43.673+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:44.010 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:43.673+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:44.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:44.009+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:44.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:44.009+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:44.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:44.009+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:44.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:44.011+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:44.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:44.254+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:44.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:44.254+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:44.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:44.254+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:44.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:44.254+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:44.968 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:44.646+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:44.968 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:44.646+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:44.968 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:44.646+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:44.968 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:44.646+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:45.239 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:44.968+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:45.239 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:44.968+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:45.239 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:44.968+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:45.240 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:44.969+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:45.531 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:04:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:04:45.342+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 13521 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:04:45.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:45.270+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:45.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:45.270+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:45.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:45.270+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:45.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:45.271+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:45.934 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:45.607+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:45.934 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:45.607+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:45.934 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:45.607+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:45.934 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:45.608+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:46.281 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:45.935+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:46.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:45.935+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:46.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:45.935+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:46.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:45.935+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:46.583 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:46.285+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:46.583 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:46.285+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:46.583 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:46.285+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:46.583 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:46.285+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:46.972 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:46.584+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:46.972 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:46.584+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:46.972 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:46.584+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:46.973 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:46.585+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:47.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:46.973+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:47.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:46.973+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:47.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:46.973+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:47.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:46.974+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:47.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:47.323+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:47.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:47.323+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:47.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:47.323+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:47.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:47.324+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:47.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:47.543+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:47.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:47.543+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:47.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:47.543+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:47.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:47.544+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:48.281 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:47.962+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:48.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:47.962+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:48.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:47.962+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:48.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:47.963+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:48.583 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:48.324+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:48.584 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:48.324+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:48.584 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:48.324+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:48.584 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:48.325+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:48.953 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:48.584+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:48.953 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:48.584+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:48.953 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:48.584+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:48.954 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:48.585+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:49.233 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:48.954+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:49.233 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:48.954+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:49.233 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:48.954+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:49.233 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:48.955+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:49.603 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:49.311+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:49.603 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:49.311+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:49.603 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:49.311+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:49.603 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:49.312+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:49.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:49.604+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:49.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:49.604+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:49.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:49.604+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:49.855 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:49.605+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:50.150 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:49.930+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:50.150 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:49.930+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:50.150 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:49.930+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:50.150 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:49.931+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:50.466 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:04:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:04:50.343+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 13611 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:04:50.467 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:50.352+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:50.467 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:50.352+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:50.467 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:50.352+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:50.467 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:50.353+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:50.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:50.596+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:50.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:50.596+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:50.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:50.596+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:50.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:50.597+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:51.281 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:50.925+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:51.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:50.925+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:51.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:50.925+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:51.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:50.926+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:51.626 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:51.351+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:51.626 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:51.351+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:51.626 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:51.351+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:51.626 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:51.352+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:51.905 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:51.626+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:51.905 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:51.626+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:51.905 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:51.626+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:51.905 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:51.627+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:52.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:51.905+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:52.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:51.905+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:52.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:51.905+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:52.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:51.906+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:52.583 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:52.305+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:52.583 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:52.305+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:52.583 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:52.305+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:52.583 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:52.306+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:52.865 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:52.583+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:52.865 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:52.583+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:52.866 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:52.583+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:52.866 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:52.584+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:53.281 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:52.866+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:53.281 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:52.866+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:53.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:52.866+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:53.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:52.867+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:53.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:53.281+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:53.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:53.282+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:53.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:53.282+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:53.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:53.282+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:53.817 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:53.549+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:53.818 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:53.549+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:53.818 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:53.549+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:53.818 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:53.550+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:54.266 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:53.818+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:54.267 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:53.818+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:54.267 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:53.818+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:54.267 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:53.819+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:54.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:54.267+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:54.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:54.267+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:54.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:54.267+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:54.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:54.268+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:55.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:54.591+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:55.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:54.591+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:55.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:54.591+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:55.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:54.591+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:55.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:54.834+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:55.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:54.834+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:55.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:54.834+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:55.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:54.834+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:55.630 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:04:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:04:55.345+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 13707 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:04:55.630 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:55.312+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:55.630 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:55.312+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:55.630 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:55.312+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:55.630 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:55.313+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:56.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:55.801+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:56.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:55.801+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:56.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:55.801+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:56.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:55.802+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:56.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:55.630+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:56.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:55.630+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:56.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:55.630+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:56.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:55.631+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:56.616 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:56.323+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:56.617 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:56.323+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:56.617 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:56.323+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:56.617 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:56.323+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:57.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:56.793+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:57.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:56.793+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:57.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:56.793+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:57.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:56.794+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:57.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:56.617+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:57.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:56.617+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:57.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:56.617+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:57.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:56.618+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:57.756 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:57.341+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:57.756 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:57.341+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:57.757 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:57.341+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:57.757 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:57.341+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:57.757 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:57.577+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:57.757 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:57.577+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:57.757 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:57.577+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:57.757 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:57.578+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:58.031 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:57.756+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:58.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:57.756+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:58.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:57.756+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:58.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:57.757+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:58.614 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:58.299+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:58.615 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:58.299+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:58.615 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:58.299+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:58.615 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:58.299+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:59.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:58.614+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:59.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:58.614+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:59.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:58.614+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:04:59.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:58.615+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:59.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:58.786+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:59.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:58.786+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:59.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:58.786+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:04:59.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:58.786+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:04:59.618 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:59.335+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:59.618 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:59.335+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:59.618 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:59.335+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:04:59.619 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:04:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:04:59.335+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:00.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:59.618+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:00.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:59.618+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:00.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:59.618+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:00.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:04:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:04:59.619+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:00.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:59.827+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:00.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:59.827+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:00.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:59.827+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:00.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:04:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:04:59.828+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:00.469 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:05:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:05:00.345+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 13802 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:05:00.469 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:00.287+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:00.470 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:00.287+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:00.470 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:00.287+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:00.470 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:00.288+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:00.781 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:00.583+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:00.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:00.583+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:00.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:00.583+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:00.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:00.584+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:01.258 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:00.874+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:01.258 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:00.874+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:01.259 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:00.874+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:01.259 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:00.875+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:01.520 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:01.258+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:01.520 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:01.258+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:01.520 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:01.258+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:01.520 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:01.259+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:01.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:01.593+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:01.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:01.593+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:01.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:01.593+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:01.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:01.593+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:02.281 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:01.840+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:02.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:01.840+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:02.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:01.840+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:02.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:01.841+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:02.639 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:02.296+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:02.639 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:02.296+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:02.639 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:02.296+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:02.639 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:02.296+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:03.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:02.840+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:03.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:02.840+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:03.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:02.840+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:03.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:02.841+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:03.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:02.638+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:03.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:02.638+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:03.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:02.638+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:03.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:02.640+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:03.602 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:03.344+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:03.602 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:03.344+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:03.602 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:03.344+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:03.603 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:03.345+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:04.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:03.841+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:04.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:03.841+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:04.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:03.841+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:04.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:03.842+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:04.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:03.602+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:04.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:03.602+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:04.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:03.602+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:04.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:03.603+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:04.592 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:04.317+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:04.592 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:04.317+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:04.592 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:04.317+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:04.593 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:04.318+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:05.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:04.650+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:05.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:04.650+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:05.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:04.650+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:05.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:04.650+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:05.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:04.809+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:05.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:04.809+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:05.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:04.809+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:05.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:04.810+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:05.646 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:05:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:05:05.347+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 13899 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:05:05.647 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:05.282+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:05.647 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:05.282+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:05.647 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:05.282+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:05.647 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:05.283+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:06.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:05.646+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:06.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:05.646+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:06.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:05.646+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:06.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:05.647+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:06.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:05.860+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:06.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:05.860+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:06.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:05.860+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:06.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:05.861+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:06.632 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:06.317+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:06.632 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:06.317+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:06.633 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:06.317+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:06.633 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:06.318+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:07.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:06.632+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:07.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:06.632+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:07.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:06.632+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:07.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:06.633+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:07.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:06.830+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:07.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:06.830+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:07.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:06.830+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:07.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:06.831+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:07.636 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:07.348+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:07.637 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:07.349+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:07.637 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:07.349+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:07.637 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:07.349+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:08.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:07.636+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:08.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:07.636+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:08.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:07.636+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:08.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:07.637+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:08.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:07.785+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:08.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:07.785+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:08.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:07.785+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:08.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:07.786+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:08.646 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:08.378+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:08.646 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:08.378+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:08.646 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:08.378+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:08.646 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:08.379+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:09.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:08.786+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:09.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:08.786+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:09.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:08.786+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:09.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:08.787+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:09.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:08.645+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:09.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:08.645+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:09.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:08.645+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:09.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:08.646+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:09.769 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:09.426+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:09.769 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:09.426+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:09.770 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:09.426+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:09.770 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:09.427+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:09.770 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:09.632+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:09.770 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:09.632+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:09.770 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:09.632+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:09.771 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:09.633+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:10.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:09.768+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:10.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:09.768+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:10.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:09.768+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:10.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:09.769+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:10.443 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:05:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:05:10.349+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 13994 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:05:10.736 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:10.443+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:10.737 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:10.443+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:10.737 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:10.443+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:10.737 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:10.444+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:10.738 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:10.633+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:10.738 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:10.633+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:10.738 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:10.633+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:10.738 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:10.634+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:11.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:10.803+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:11.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:10.803+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:11.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:10.803+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:11.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:10.804+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:11.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:11.428+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:11.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:11.428+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:11.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:11.428+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:11.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:11.429+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:11.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:11.668+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:11.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:11.668+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:11.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:11.668+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:11.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:11.669+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:12.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:11.840+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:12.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:11.840+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:12.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:11.840+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:12.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:11.841+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:12.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:12.628+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:12.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:12.628+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:12.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:12.628+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:12.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:12.628+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:12.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:12.476+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:12.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:12.476+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:12.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:12.476+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:12.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:12.477+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:13.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:12.864+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:13.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:12.864+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:13.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:12.864+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:13.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:12.865+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:13.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:13.458+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:13.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:13.458+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:13.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:13.458+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:13.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:13.459+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:13.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:13.640+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:13.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:13.640+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:13.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:13.640+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:13.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:13.641+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:14.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:13.861+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:14.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:13.861+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:14.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:13.861+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:14.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:13.861+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:14.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:14.648+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:14.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:14.648+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:14.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:14.648+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:14.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:14.649+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:14.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:14.490+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:14.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:14.490+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:14.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:14.490+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:14.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:14.491+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:15.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:14.904+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:15.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:14.904+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:15.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:14.904+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:15.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:14.905+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:15.536 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:05:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:05:15.350+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 14090 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:05:15.950 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:15.536+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:15.950 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:15.536+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:15.951 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:15.536+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:15.951 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:15.537+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:15.951 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:15.635+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:15.951 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:15.635+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:15.952 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:15.635+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:15.952 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:15.636+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:16.281 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:15.950+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:16.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:15.950+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:16.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:15.950+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:16.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:15.951+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:16.998 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:16.598+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:16.999 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:16.598+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:16.999 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:16.598+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:16.999 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:16.599+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:16.999 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:16.549+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:17.000 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:16.549+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:17.000 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:16.549+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:17.000 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:16.550+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:17.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:16.998+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:17.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:16.998+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:17.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:16.998+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:17.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:16.999+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:18.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:17.547+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:18.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:17.547+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:18.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:17.547+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:18.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:17.547+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:18.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:17.620+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:18.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:17.620+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:18.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:17.620+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:18.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:17.621+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:18.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:18.045+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:18.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:18.045+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:18.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:18.045+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:18.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:18.046+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:19.007 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:18.551+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:19.008 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:18.551+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:19.008 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:18.551+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:19.008 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:18.551+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:19.008 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:18.591+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:19.008 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:18.591+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:19.009 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:18.591+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:19.009 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:18.591+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:19.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:19.006+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:19.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:19.006+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:19.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:19.006+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:19.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:19.008+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:19.853 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:19.596+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:19.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:19.596+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:19.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:19.596+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:19.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:19.597+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:19.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:19.590+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:19.855 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:19.590+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:19.855 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:19.590+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:19.855 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:19.591+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:20.151 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:19.966+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:20.152 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:19.966+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:20.152 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:19.966+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:20.152 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:19.966+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:20.477 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:05:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:05:20.352+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 14184 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:05:20.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:20.576+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:20.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:20.576+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:20.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:20.576+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:20.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:20.577+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:20.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:20.631+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:20.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:20.631+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:20.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:20.631+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:20.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:20.632+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:21.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:20.952+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:21.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:20.952+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:21.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:20.952+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:21.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:20.954+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:21.981 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:21.587+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:21.981 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:21.587+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:21.981 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:21.587+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:21.981 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:21.588+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:21.982 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:21.648+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:21.982 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:21.648+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:21.982 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:21.648+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:21.982 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:21.648+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:22.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:21.980+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:22.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:21.980+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:22.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:21.980+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:22.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:21.981+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:22.995 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:22.609+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:22.995 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:22.609+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:22.995 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:22.609+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:22.995 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:22.610+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:22.996 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:22.618+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:22.996 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:22.618+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:22.996 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:22.618+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:22.996 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:22.619+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:23.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:22.994+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:23.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:22.994+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:23.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:22.994+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:23.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:22.995+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:23.971 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:23.603+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:23.971 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:23.603+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:23.971 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:23.603+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:23.971 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:23.604+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:23.972 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:23.583+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:23.972 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:23.583+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:23.972 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:23.583+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:23.972 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:23.584+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:24.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:23.970+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:24.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:23.970+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:24.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:23.970+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:24.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:23.971+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:24.925 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:24.611+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:24.925 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:24.611+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:24.925 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:24.611+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:24.926 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:24.611+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:24.926 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:24.621+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:24.926 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:24.621+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:24.926 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:24.621+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:24.927 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:24.622+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:25.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:24.924+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:25.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:24.924+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:25.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:24.924+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:25.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:24.925+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:25.622 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:05:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:05:25.353+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 14275 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:05:25.952 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:25.622+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:25.952 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:25.622+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:25.952 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:25.622+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:25.952 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:25.623+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:25.952 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:25.660+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:25.953 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:25.660+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:25.953 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:25.660+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:25.953 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:25.661+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:26.281 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:25.952+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:26.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:25.952+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:26.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:25.952+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:26.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:25.952+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:26.981 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:26.629+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:26.981 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:26.629+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:26.982 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:26.629+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:26.982 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:26.629+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:26.982 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:26.625+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:26.982 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:26.625+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:26.982 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:26.625+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:26.982 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:26.626+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:27.281 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:26.981+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:27.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:26.981+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:27.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:26.981+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:27.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:26.981+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:27.949 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:27.601+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:27.949 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:27.601+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:27.950 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:27.601+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:27.950 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:27.602+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:27.950 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:27.611+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:27.950 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:27.611+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:27.950 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:27.611+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:27.951 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:27.612+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:28.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:27.948+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:28.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:27.948+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:28.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:27.948+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:28.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:27.949+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:28.940 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:28.635+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:28.941 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:28.635+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:28.941 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:28.635+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:28.941 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:28.636+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:28.941 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:28.625+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:28.941 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:28.625+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:28.941 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:28.625+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:28.941 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:28.626+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:29.281 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:28.940+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:29.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:28.940+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:29.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:28.940+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:29.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:28.941+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:30.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:29.661+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:30.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:29.661+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:30.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:29.661+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:30.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:29.662+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:30.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:29.658+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:30.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:29.658+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:30.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:29.658+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:30.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:29.659+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:30.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:29.894+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:30.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:29.894+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:30.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:29.894+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:30.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:29.895+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:30.477 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:05:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:05:30.355+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 14362 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:05:30.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:30.682+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:30.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:30.682+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:30.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:30.682+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:30.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:30.683+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:30.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:30.663+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:30.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:30.664+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:30.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:30.664+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:30.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:30.664+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:31.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:30.914+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:31.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:30.914+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:31.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:30.914+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:31.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:30.914+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:32.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:31.873+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:32.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:31.873+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:32.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:31.873+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:32.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:31.874+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:32.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:31.714+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:32.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:31.714+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:32.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:31.714+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:32.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:31.715+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:32.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:31.655+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:32.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:31.655+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:32.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:31.655+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:32.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:31.655+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:33.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:32.748+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:33.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:32.748+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:33.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:32.748+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:33.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:32.748+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:33.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:32.856+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:33.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:32.856+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:33.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:32.856+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:33.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:32.857+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:33.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:32.612+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:33.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:32.612+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:33.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:32.612+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:33.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:32.613+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:33.881 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:33.764+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:33.882 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:33.764+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:33.882 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:33.764+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:33.882 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:33.766+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:33.882 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:33.601+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:33.882 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:33.601+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:33.883 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:33.601+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:33.883 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:33.602+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:34.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:33.878+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:34.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:33.878+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:34.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:33.878+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:34.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:33.879+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:34.876 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:34.724+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:34.876 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:34.724+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:34.877 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:34.724+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:34.877 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:34.725+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:34.877 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:34.581+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:34.877 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:34.581+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:34.877 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:34.581+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:34.877 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:34.582+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:35.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:34.876+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:35.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:34.876+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:35.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:34.876+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:35.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:34.876+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:35.597 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:05:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:05:35.356+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 14450 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:05:36.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:35.840+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:36.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:35.840+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:36.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:35.840+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:36.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:35.840+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:36.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:35.707+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:36.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:35.707+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:36.035 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:35.707+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:36.035 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:35.708+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:36.035 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:35.597+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:36.035 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:35.597+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:36.035 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:35.597+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:36.036 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:35.598+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:37.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:36.671+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:37.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:36.671+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:37.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:36.671+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:37.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:36.671+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:37.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:36.636+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:37.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:36.636+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:37.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:36.636+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:37.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:36.637+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:37.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:36.879+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:37.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:36.879+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:37.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:36.879+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:37.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:36.880+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:38.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:37.632+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:38.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:37.632+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:38.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:37.632+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:38.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:37.633+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:38.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:37.639+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:38.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:37.639+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:38.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:37.639+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:38.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:37.641+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:38.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:37.860+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:38.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:37.860+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:38.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:37.860+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:38.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:37.861+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:38.657 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:38.631+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:38.657 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:38.631+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:38.657 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:38.631+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:38.657 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:38.632+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:38.657+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:38.657+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:38.657+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:38.658+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:39.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:38.840+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:39.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:38.840+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:39.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:38.840+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:39.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:38.841+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:39.831 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:39.829+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:39.831 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:39.664+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:39.831 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:39.664+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:39.831 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:39.664+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:39.831 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:39.665+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:39.832 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:39.607+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:39.832 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:39.607+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:39.832 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:39.607+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:39.832 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:39.608+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:40.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:39.829+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:40.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:39.829+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:40.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:39.830+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:40.481 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:05:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:05:40.358+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 14537 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:05:40.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:40.626+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:40.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:40.626+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:40.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:40.626+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:40.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:40.627+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:40.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:40.646+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:40.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:40.646+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:40.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:40.646+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:40.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:40.647+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:41.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:40.842+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:41.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:40.842+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:41.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:40.842+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:41.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:40.843+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:41.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:41.654+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:41.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:41.654+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:41.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:41.654+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:41.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:41.655+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:41.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:41.639+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:41.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:41.639+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:41.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:41.639+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:41.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:41.639+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:42.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:41.853+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:42.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:41.853+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:42.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:41.853+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:42.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:41.853+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:42.889 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:42.629+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:42.890 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:42.629+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:42.890 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:42.629+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:42.890 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:42.630+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:42.891 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:42.611+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:42.891 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:42.611+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:42.891 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:42.611+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:42.891 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:42.613+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:43.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:42.889+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:43.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:42.889+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:43.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:42.889+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:43.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:42.890+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:43.844 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:43.615+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:43.845 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:43.615+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:43.845 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:43.615+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:43.845 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:43.616+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:43.845 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:43.597+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:43.846 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:43.597+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:43.846 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:43.597+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:43.846 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:43.598+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:44.281 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:43.844+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:44.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:43.844+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:44.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:43.844+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:44.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:43.845+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:44.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:44.625+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:44.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:44.625+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:44.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:44.625+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:44.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:44.626+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:44.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:44.582+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:44.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:44.582+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:44.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:44.582+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:44.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:44.582+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:45.281 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:44.810+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:45.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:44.810+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:45.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:44.810+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:45.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:44.811+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:45.609 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:05:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:05:45.360+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 14627 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:05:46.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:45.648+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:46.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:45.648+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:46.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:45.648+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:46.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:45.649+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:46.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:45.609+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:46.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:45.609+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:46.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:45.609+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:46.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:45.610+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:46.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:45.840+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:46.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:45.840+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:46.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:45.840+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:46.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:45.841+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:47.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:46.657+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:47.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:46.657+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:47.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:46.657+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:47.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:46.657+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:47.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:46.870+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:47.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:46.870+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:47.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:46.870+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:47.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:46.871+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:47.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:46.634+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:47.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:46.634+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:47.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:46.634+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:47.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:46.634+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:47.855 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:47.681+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:47.855 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:47.681+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:47.855 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:47.681+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:47.855 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:47.682+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:47.856 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:47.621+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:47.856 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:47.621+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:47.856 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:47.621+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:47.856 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:47.622+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:48.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:47.855+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:48.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:47.855+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:48.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:47.855+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:48.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:47.855+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:49.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:48.726+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:49.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:48.726+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:49.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:48.726+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:49.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:48.727+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:49.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:48.865+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:49.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:48.865+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:49.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:48.865+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:49.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:48.865+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:49.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:48.670+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:49.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:48.670+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:49.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:48.670+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:49.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:48.671+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:50.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:49.838+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:50.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:49.838+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:50.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:49.838+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:50.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:49.839+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:50.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:49.683+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:50.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:49.683+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:50.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:49.683+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:50.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:49.684+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:50.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:49.695+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:50.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:49.695+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:50.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:49.695+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:50.035 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:49.695+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:50.486 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:05:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:05:50.361+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 14715 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:05:50.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:50.693+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:50.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:50.693+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:50.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:50.693+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:50.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:50.694+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:50.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:50.673+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:50.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:50.673+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:50.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:50.673+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:50.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:50.674+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:51.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:50.834+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:51.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:50.834+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:51.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:50.834+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:51.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:50.835+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:52.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:51.649+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:52.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:51.649+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:52.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:51.649+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:52.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:51.650+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:52.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:51.868+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:52.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:51.868+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:52.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:51.868+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:52.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:51.869+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:52.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:51.708+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:52.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:51.708+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:52.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:51.708+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:52.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:51.709+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:53.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:52.661+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:53.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:52.661+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:53.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:52.661+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:53.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:52.662+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:53.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:52.736+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:53.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:52.736+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:53.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:52.736+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:53.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:52.737+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:53.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:52.894+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:53.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:52.894+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:53.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:52.894+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:53.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:52.894+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:53.867 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:53.762+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:53.868 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:53.762+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:53.868 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:53.762+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:53.868 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:53.763+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:53.868 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:53.621+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:53.869 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:53.621+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:53.869 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:53.621+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:53.869 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:53.622+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:54.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:53.867+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:54.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:53.867+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:54.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:53.867+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:54.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:53.868+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:55.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:54.637+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:55.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:54.637+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:55.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:54.637+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:55.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:54.638+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:55.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:54.772+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:55.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:54.772+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:55.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:54.772+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:55.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:54.773+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:55.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:54.827+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:55.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:54.827+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:55.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:54.827+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:55.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:54.828+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:55.657 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:05:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:05:55.363+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 14804 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:05:56.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:55.657+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:56.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:55.657+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:56.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:55.657+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:56.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:55.658+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:56.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:55.771+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:56.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:55.771+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:56.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:55.771+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:56.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:55.771+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:56.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:55.780+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:56.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:55.780+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:56.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:55.780+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:56.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:55.781+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:57.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:56.703+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:57.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:56.703+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:57.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:56.703+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:57.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:56.703+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:57.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:56.763+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:57.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:56.763+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:57.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:56.763+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:57.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:56.764+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:57.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:56.764+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:57.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:56.764+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:57.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:56.764+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:57.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:56.765+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:58.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:57.745+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:58.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:57.745+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:58.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:57.745+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:58.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:57.746+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:58.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:57.656+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:58.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:57.656+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:58.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:57.656+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:58.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:57.657+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:58.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:57.716+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:58.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:57.716+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:58.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:57.716+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:58.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:57.716+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:59.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:58.766+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:59.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:58.766+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:59.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:58.766+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:59.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:58.767+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:59.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:58.618+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:59.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:58.618+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:59.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:58.618+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:59.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:58.619+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:59.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:58.685+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:59.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:58.685+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:59.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:58.685+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:59.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:58.686+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:59.889 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:59.743+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:59.889 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:59.743+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:59.890 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:59.743+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:05:59.890 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:05:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:05:59.744+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:59.890 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:59.609+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:59.890 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:59.609+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:59.890 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:59.609+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:05:59.891 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:05:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:05:59.610+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:05:59.891 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:59.720+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:59.891 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:59.720+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:59.891 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:59.720+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:05:59.891 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:05:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:05:59.721+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:00.488 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:06:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:06:00.365+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 14891 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:06:00.778 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:00.732+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:00.778 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:00.732+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:00.779 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:00.732+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:00.779 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:00.732+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:00.779 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:00.588+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:00.779 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:00.588+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:00.780 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:00.588+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:00.780 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:00.589+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:01.031 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:00.777+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:01.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:00.777+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:01.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:00.777+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:01.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:00.778+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:02.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:01.548+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:02.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:01.548+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:02.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:01.548+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:02.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:01.549+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:02.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:01.758+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:02.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:01.758+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:02.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:01.758+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:02.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:01.759+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:02.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:01.756+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:02.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:01.756+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:02.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:01.756+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:02.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:01.756+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:02.801 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:02.535+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:02.801 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:02.535+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:02.801 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:02.535+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:02.801 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:02.536+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:02.801 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:02.718+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:02.802 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:02.718+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:02.802 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:02.718+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:02.802 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:02.719+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:02.816 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-24T10:06:02.816 WARNING:tasks.kclient:Ordinary umount failed, forcing... 2024-03-24T10:06:02.816 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T10:06:02.817 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:06:02.832 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-24T10:06:02.832 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:06:02.889 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:06:03.281 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:02.800+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:03.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:02.800+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:03.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:02.800+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:03.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:02.801+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:03.818 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:03.700+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:03.819 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:03.700+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:03.819 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:03.700+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:03.819 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:03.701+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:03.819 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:03.565+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:03.819 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:03.565+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:03.819 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:03.565+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:03.820 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:03.566+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:04.281 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:03.818+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:04.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:03.818+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:04.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:03.818+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:04.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:03.819+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:05.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:04.575+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:05.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:04.575+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:05.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:04.575+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:05.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:04.576+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:05.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:04.800+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:05.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:04.800+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:05.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:04.800+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:05.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:04.801+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:05.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:04.744+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:05.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:04.744+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:05.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:04.744+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:05.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:04.745+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:05.616 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:06:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:06:05.366+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 14982 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:06:05.617 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:05.616+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:05.617 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:05.616+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:05.617 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:05.616+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:05.915 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:05.617+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:05.916 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:05.744+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:05.916 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:05.744+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:05.916 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:05.744+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:05.916 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:05.745+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:05.916 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:05.810+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:05.916 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:05.810+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:05.916 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:05.810+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:05.917 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:05.811+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:07.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:06.789+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:07.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:06.789+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:07.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:06.789+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:07.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:06.790+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:07.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:06.605+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:07.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:06.605+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:07.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:06.605+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:07.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:06.606+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:07.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:06.725+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:07.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:06.725+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:07.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:06.725+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:07.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:06.725+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:08.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:07.786+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:08.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:07.786+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:08.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:07.786+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:08.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:07.787+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:08.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:07.561+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:08.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:07.561+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:08.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:07.561+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:08.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:07.562+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:08.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:07.695+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:08.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:07.695+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:08.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:07.695+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:08.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:07.696+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:09.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:08.831+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:09.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:08.831+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:09.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:08.831+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:09.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:08.831+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:09.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:08.604+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:09.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:08.604+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:09.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:08.604+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:09.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:08.605+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:09.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:08.684+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:09.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:08.684+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:09.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:08.684+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:09.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:08.685+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:09.852 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:09.587+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:09.852 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:09.587+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:09.853 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:09.587+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:09.853 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:09.588+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:09.853 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:09.662+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:09.853 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:09.662+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:09.853 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:09.662+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:09.853 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:09.662+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:10.155 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:09.853+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:10.155 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:09.853+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:10.156 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:09.853+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:10.156 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:09.854+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:10.492 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:06:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:06:10.369+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 15069 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:06:10.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:10.656+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:10.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:10.656+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:10.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:10.656+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:10.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:10.656+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:10.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:10.559+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:10.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:10.559+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:10.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:10.559+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:10.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:10.560+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:11.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:10.823+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:11.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:10.823+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:11.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:10.823+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:11.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:10.824+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:11.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:11.652+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:11.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:11.652+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:11.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:11.652+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:11.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:11.653+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:11.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:11.524+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:11.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:11.524+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:11.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:11.524+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:11.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:11.525+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:12.176 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:11.860+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:12.177 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:11.860+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:12.177 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:11.860+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:12.177 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:11.860+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:12.899 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:12.613+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:12.900 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:12.613+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:12.900 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:12.613+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:12.900 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:12.614+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:12.900 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:12.548+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:12.901 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:12.548+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:12.901 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:12.548+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:12.901 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:12.548+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:13.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:12.900+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:13.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:12.900+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:13.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:12.900+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:13.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:12.901+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:13.864 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:13.575+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:13.865 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:13.575+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:13.865 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:13.575+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:13.865 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:13.576+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:13.865 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:13.550+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:13.866 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:13.550+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:13.866 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:13.550+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:13.866 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:13.550+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:14.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:13.864+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:14.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:13.864+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:14.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:13.864+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:14.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:13.865+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:14.853 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:14.537+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:14.853 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:14.537+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:14.853 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:14.537+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:14.853 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:14.538+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:14.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:14.599+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:14.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:14.599+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:14.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:14.599+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:14.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:14.599+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:15.248 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:14.853+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:15.249 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:14.853+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:15.249 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:14.853+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:15.249 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:14.854+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:15.512 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:06:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:06:15.371+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 15159 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:06:15.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:15.513+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:15.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:15.513+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:15.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:15.513+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:15.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:15.514+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:15.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:15.618+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:15.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:15.618+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:15.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:15.618+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:15.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:15.619+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:16.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:15.818+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:16.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:15.818+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:16.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:15.818+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:16.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:15.818+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:16.781 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:16.657+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:16.781 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:16.657+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:16.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:16.657+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:16.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:16.658+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:16.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:16.482+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:16.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:16.482+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:16.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:16.482+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:16.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:16.482+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:17.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:16.782+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:17.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:16.782+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:17.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:16.782+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:17.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:16.783+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:18.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:17.672+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:18.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:17.672+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:18.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:17.672+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:18.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:17.672+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:18.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:17.532+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:18.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:17.532+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:18.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:17.532+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:18.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:17.533+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:18.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:17.759+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:18.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:17.759+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:18.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:17.759+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:18.035 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:17.760+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:19.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:18.629+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:19.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:18.629+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:19.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:18.629+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:19.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:18.630+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:19.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:18.711+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:19.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:18.711+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:19.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:18.711+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:19.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:18.712+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:19.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:18.563+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:19.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:18.563+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:19.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:18.563+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:19.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:18.564+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:19.853 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:19.597+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:19.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:19.597+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:19.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:19.597+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:19.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:19.598+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:19.855 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:19.709+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:19.855 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:19.709+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:19.855 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:19.709+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:19.855 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:19.709+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:19.856 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:19.621+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:19.856 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:19.621+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:19.856 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:19.621+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:19.856 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:19.622+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:20.496 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:06:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:06:20.371+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 15246 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:06:20.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:20.730+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:20.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:20.730+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:20.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:20.730+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:20.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:20.730+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:20.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:20.602+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:20.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:20.603+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:20.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:20.603+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:20.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:20.603+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:20.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:20.621+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:20.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:20.621+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:20.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:20.621+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:20.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:20.622+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:21.774 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:21.593+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:21.775 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:21.593+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:21.775 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:21.593+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:21.775 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:21.593+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:21.775 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:21.623+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:21.776 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:21.623+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:21.776 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:21.623+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:21.776 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:21.623+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:22.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:21.774+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:22.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:21.774+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:22.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:21.774+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:22.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:21.775+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:23.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:22.612+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:23.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:22.613+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:23.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:22.613+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:23.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:22.613+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:23.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:22.735+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:23.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:22.735+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:23.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:22.735+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:23.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:22.736+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:23.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:22.628+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:23.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:22.628+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:23.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:22.628+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:23.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:22.629+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:24.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:23.652+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:24.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:23.652+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:24.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:23.652+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:24.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:23.653+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:24.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:23.691+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:24.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:23.691+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:24.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:23.691+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:24.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:23.692+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:24.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:23.639+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:24.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:23.639+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:24.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:23.639+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:24.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:23.639+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:24.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:24.623+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:24.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:24.623+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:24.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:24.623+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:24.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:24.624+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:24.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:24.647+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:24.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:24.647+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:24.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:24.647+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:24.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:24.648+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:24.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:24.675+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:24.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:24.675+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:24.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:24.675+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:24.785 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:24.676+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:25.629 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:06:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:06:25.373+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 15337 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:06:25.629 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:25.606+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:25.629 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:25.606+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:25.630 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:25.606+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:25.630 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:25.607+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:26.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:25.628+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:26.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:25.628+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:26.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:25.628+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:26.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:25.629+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:26.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:25.666+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:26.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:25.666+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:26.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:25.666+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:26.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:25.667+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:27.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:26.637+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:27.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:26.637+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:27.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:26.637+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:27.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:26.638+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:27.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:26.569+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:27.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:26.569+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:27.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:26.569+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:27.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:26.570+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:27.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:26.689+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:27.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:26.689+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:27.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:26.689+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:27.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:26.689+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:28.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:27.576+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:28.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:27.576+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:28.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:27.576+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:28.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:27.577+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:28.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:27.617+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:28.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:27.617+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:28.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:27.617+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:28.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:27.618+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:28.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:27.660+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:28.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:27.660+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:28.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:27.660+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:28.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:27.661+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:29.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:28.623+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:29.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:28.623+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:29.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:28.623+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:29.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:28.624+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:29.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:28.584+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:29.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:28.584+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:29.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:28.584+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:29.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:28.585+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:29.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:28.554+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:29.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:28.554+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:29.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:28.554+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:29.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:28.555+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:29.853 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:29.639+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:29.853 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:29.639+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:29.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:29.639+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:29.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:29.639+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:29.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:29.537+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:29.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:29.537+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:29.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:29.537+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:29.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:29.538+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:29.855 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:29.564+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:29.855 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:29.564+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:29.855 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:29.564+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:29.855 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:29.565+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:30.499 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:06:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:06:30.375+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 15423 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:06:30.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:30.592+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:30.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:30.592+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:30.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:30.592+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:30.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:30.593+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:30.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:30.575+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:30.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:30.575+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:30.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:30.575+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:30.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:30.576+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:30.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:30.602+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:30.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:30.602+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:30.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:30.602+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:30.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:30.603+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:32.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:31.558+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:32.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:31.558+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:32.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:31.558+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:32.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:31.559+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:32.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:31.629+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:32.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:31.629+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:32.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:31.629+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:32.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:31.630+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:32.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:31.627+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:32.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:31.627+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:32.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:31.627+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:32.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:31.627+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:32.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:32.628+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:32.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:32.628+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:32.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:32.628+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:32.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:32.629+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:32.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:32.514+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:32.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:32.514+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:32.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:32.514+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:32.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:32.515+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:32.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:32.610+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:32.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:32.610+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:32.785 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:32.610+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:32.785 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:32.611+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:34.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:33.601+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:34.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:33.601+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:34.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:33.601+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:34.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:33.601+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:34.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:33.540+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:34.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:33.540+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:34.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:33.540+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:34.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:33.541+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:34.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:33.638+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:34.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:33.638+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:34.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:33.638+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:34.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:33.638+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:35.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:34.640+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:35.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:34.640+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:35.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:34.640+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:35.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:34.641+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:35.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:34.672+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:35.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:34.672+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:35.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:34.672+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:35.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:34.673+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:35.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:34.550+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:35.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:34.550+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:35.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:34.550+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:35.035 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:34.551+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:35.663 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:06:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:06:35.376+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 15512 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:06:35.663 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:35.583+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:35.663 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:35.583+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:35.663 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:35.583+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:35.664 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:35.584+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:36.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:35.720+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:36.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:35.720+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:36.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:35.720+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:36.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:35.720+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:36.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:35.662+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:36.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:35.662+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:36.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:35.662+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:36.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:35.663+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:37.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:36.562+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:37.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:36.563+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:37.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:36.563+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:37.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:36.563+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:37.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:36.764+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:37.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:36.764+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:37.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:36.764+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:37.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:36.765+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:37.035 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:36.641+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:37.035 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:36.641+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:37.035 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:36.641+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:37.035 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:36.642+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:38.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:37.592+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:38.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:37.592+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:38.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:37.592+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:38.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:37.593+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:38.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:37.613+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:38.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:37.613+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:38.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:37.613+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:38.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:37.614+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:38.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:37.735+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:38.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:37.735+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:38.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:37.735+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:38.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:37.735+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:39.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:38.616+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:39.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:38.616+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:39.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:38.616+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:39.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:38.616+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:39.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:38.708+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:39.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:38.708+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:39.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:38.708+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:39.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:38.708+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:39.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:38.662+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:39.035 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:38.662+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:39.035 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:38.662+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:39.035 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:38.663+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:39.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:39.732+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:39.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:39.732+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:39.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:39.732+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:39.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:39.733+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:39.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:39.599+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:39.855 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:39.599+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:39.855 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:39.599+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:39.855 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:39.599+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:39.855 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:39.707+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:39.855 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:39.707+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:39.855 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:39.707+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:39.855 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:39.708+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:40.502 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:06:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:06:40.378+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 15598 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:06:40.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:40.695+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:40.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:40.695+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:40.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:40.695+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:40.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:40.696+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:40.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:40.690+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:40.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:40.690+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:40.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:40.690+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:40.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:40.691+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:40.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:40.563+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:40.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:40.563+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:40.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:40.563+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:40.785 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:40.564+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:42.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:41.740+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:42.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:41.740+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:42.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:41.740+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:42.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:41.741+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:42.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:41.673+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:42.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:41.673+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:42.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:41.673+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:42.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:41.673+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:42.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:41.604+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:42.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:41.604+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:42.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:41.604+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:42.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:41.605+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:42.896 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:42.746+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:42.897 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:42.746+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:42.897 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:42.746+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:42.897 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:42.747+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:42.897 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:42.630+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:42.898 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:42.630+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:42.898 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:42.630+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:42.898 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:42.631+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:42.899 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:42.707+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:42.899 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:42.707+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:42.899 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:42.707+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:42.899 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:42.708+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:44.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:43.777+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:44.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:43.777+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:44.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:43.778+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:44.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:43.779+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:44.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:43.684+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:44.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:43.684+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:44.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:43.684+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:44.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:43.684+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:44.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:43.592+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:44.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:43.592+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:44.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:43.592+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:44.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:43.593+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:45.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:44.591+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:45.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:44.591+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:45.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:44.591+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:45.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:44.591+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:45.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:44.680+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:45.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:44.680+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:45.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:44.680+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:45.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:44.681+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:45.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:44.792+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:45.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:44.792+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:45.035 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:44.792+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:45.035 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:44.793+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:45.728 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:06:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:06:45.380+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 15687 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:06:45.729 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:45.582+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:45.729 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:45.582+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:45.729 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:45.582+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:45.729 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:45.583+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:46.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:45.826+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:46.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:45.826+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:46.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:45.826+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:46.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:45.827+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:46.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:45.728+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:46.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:45.728+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:46.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:45.728+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:46.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:45.729+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:47.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:46.721+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:47.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:46.721+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:47.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:46.721+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:47.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:46.722+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:47.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:46.577+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:47.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:46.577+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:47.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:46.577+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:47.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:46.578+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:47.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:46.787+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:47.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:46.787+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:47.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:46.787+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:47.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:46.788+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:48.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:47.590+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:48.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:47.590+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:48.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:47.590+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:48.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:47.591+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:48.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:47.746+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:48.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:47.746+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:48.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:47.746+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:48.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:47.747+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:48.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:47.684+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:48.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:47.684+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:48.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:47.684+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:48.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:47.685+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:49.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:48.705+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:49.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:48.705+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:49.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:48.705+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:49.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:48.705+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:49.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:48.575+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:49.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:48.575+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:49.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:48.575+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:49.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:48.575+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:49.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:48.787+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:49.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:48.787+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:49.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:48.787+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:49.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:48.788+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:49.853 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:49.730+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:49.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:49.730+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:49.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:49.730+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:49.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:49.731+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:49.855 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:49.533+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:49.855 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:49.533+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:49.855 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:49.533+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:49.855 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:49.535+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:49.856 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:49.745+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:49.856 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:49.745+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:49.856 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:49.745+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:49.856 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:49.746+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:50.506 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:06:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:06:50.382+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 15773 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:06:50.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:50.754+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:50.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:50.754+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:50.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:50.754+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:50.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:50.755+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:50.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:50.573+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:50.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:50.573+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:50.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:50.573+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:50.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:50.575+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:50.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:50.685+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:50.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:50.685+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:50.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:50.685+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:50.785 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:50.686+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:52.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:51.763+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:52.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:51.763+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:52.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:51.763+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:52.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:51.764+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:52.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:51.598+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:52.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:51.598+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:52.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:51.598+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:52.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:51.599+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:52.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:51.654+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:52.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:51.654+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:52.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:51.654+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:52.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:51.655+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:53.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:52.670+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:53.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:52.670+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:53.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:52.670+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:53.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:52.671+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:53.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:52.606+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:53.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:52.606+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:53.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:52.606+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:53.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:52.607+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:53.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:52.721+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:53.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:52.721+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:53.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:52.721+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:53.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:52.722+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:54.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:53.673+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:54.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:53.673+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:54.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:53.673+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:54.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:53.673+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:54.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:53.634+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:54.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:53.634+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:54.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:53.634+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:54.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:53.635+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:54.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:53.695+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:54.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:53.695+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:54.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:53.695+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:54.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:53.696+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:55.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:54.660+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:55.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:54.660+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:55.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:54.660+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:55.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:54.661+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:55.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:54.663+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:55.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:54.663+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:55.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:54.663+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:55.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:54.664+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:55.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:54.652+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:55.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:54.652+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:55.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:54.652+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:55.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:54.653+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:55.508 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:06:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:06:55.383+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 15863 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:06:55.781 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:55.676+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:55.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:55.676+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:55.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:55.676+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:55.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:55.677+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:55.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:55.677+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:55.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:55.677+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:55.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:55.677+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:55.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:55.677+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:55.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:55.656+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:55.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:55.656+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:55.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:55.656+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:55.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:55.656+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:57.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:56.685+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:57.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:56.685+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:57.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:56.685+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:57.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:56.686+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:57.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:56.617+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:57.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:56.617+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:57.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:56.617+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:57.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:56.618+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:57.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:56.718+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:57.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:56.718+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:57.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:56.718+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:57.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:56.719+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:58.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:57.652+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:58.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:57.652+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:58.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:57.652+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:58.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:57.653+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:58.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:57.657+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:58.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:57.657+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:58.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:57.657+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:58.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:57.658+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:58.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:57.684+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:58.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:57.684+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:58.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:57.684+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:58.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:57.685+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:59.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:58.678+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:59.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:58.678+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:59.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:58.678+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:59.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:58.679+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:59.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:58.660+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:59.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:58.660+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:59.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:58.660+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:59.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:58.660+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:59.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:58.608+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:59.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:58.608+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:59.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:58.608+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:59.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:58.609+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:59.853 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:59.565+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:59.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:59.565+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:59.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:59.565+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:06:59.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:06:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:06:59.566+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:59.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:59.720+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:59.855 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:59.720+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:59.855 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:59.720+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:06:59.855 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:06:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:06:59.721+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:06:59.855 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:59.692+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:59.855 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:59.692+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:59.856 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:59.692+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:06:59.856 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:06:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:06:59.693+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:00.507 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:07:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:07:00.385+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 15947 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:07:00.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:00.711+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:00.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:00.711+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:00.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:00.711+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:00.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:00.712+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:00.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:00.614+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:00.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:00.614+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:00.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:00.614+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:00.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:00.615+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:00.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:00.696+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:00.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:00.696+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:00.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:00.696+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:00.785 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:00.697+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:02.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:01.634+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:02.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:01.634+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:02.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:01.634+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:02.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:01.635+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:02.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:01.699+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:02.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:01.699+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:02.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:01.699+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:02.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:01.700+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:02.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:01.699+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:02.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:01.699+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:02.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:01.699+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:02.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:01.700+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:03.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:02.678+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:03.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:02.678+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:03.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:02.678+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:03.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:02.679+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:03.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:02.671+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:03.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:02.671+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:03.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:02.671+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:03.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:02.671+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:03.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:02.723+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:03.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:02.723+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:03.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:02.723+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:03.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:02.724+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:04.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:03.680+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:04.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:03.680+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:04.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:03.680+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:04.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:03.681+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:04.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:03.705+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:04.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:03.705+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:04.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:03.705+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:04.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:03.706+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:04.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:03.653+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:04.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:03.653+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:04.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:03.653+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:04.035 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:03.653+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:05.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:04.658+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:05.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:04.658+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:05.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:04.658+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:05.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:04.659+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:05.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:04.671+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:05.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:04.671+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:05.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:04.671+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:05.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:04.672+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:05.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:04.684+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:05.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:04.684+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:05.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:04.684+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:05.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:04.685+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:05.641 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:07:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:07:05.387+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 16035 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:07:06.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:05.640+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:06.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:05.640+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:06.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:05.641+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:06.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:05.641+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:06.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:05.696+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:06.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:05.696+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:06.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:05.696+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:06.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:05.697+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:06.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:05.662+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:06.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:05.662+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:06.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:05.662+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:06.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:05.662+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:06.680 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:06.615+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:06.680 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:06.615+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:06.680 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:06.615+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:06.680 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:06.615+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:07.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:06.701+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:07.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:06.701+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:07.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:06.701+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:07.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:06.702+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:07.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:06.679+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:07.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:06.679+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:07.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:06.679+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:07.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:06.681+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:07.725 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:07.673+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:07.726 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:07.673+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:07.726 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:07.673+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:07.726 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:07.674+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:07.726 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:07.615+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:07.726 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:07.615+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:07.726 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:07.615+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:07.726 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:07.616+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:08.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:07.725+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:08.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:07.725+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:08.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:07.725+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:08.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:07.726+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:09.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:08.579+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:09.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:08.580+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:09.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:08.580+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:09.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:08.580+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:09.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:08.644+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:09.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:08.644+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:09.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:08.644+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:09.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:08.645+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:09.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:08.678+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:09.035 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:08.678+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:09.035 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:08.678+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:09.035 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:08.679+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:09.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:09.666+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:09.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:09.666+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:09.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:09.666+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:09.855 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:09.666+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:09.855 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:09.602+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:09.855 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:09.602+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:09.855 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:09.602+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:09.856 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:09.603+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:09.856 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:09.630+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:09.856 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:09.630+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:09.856 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:09.630+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:09.856 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:09.631+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:10.413 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:07:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:07:10.389+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 16120 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:07:10.702 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:10.652+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:10.702 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:10.652+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:10.702 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:10.652+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:10.702 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:10.653+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:10.703 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:10.593+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:10.703 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:10.593+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:10.703 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:10.593+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:10.703 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:10.593+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:11.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:10.702+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:11.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:10.702+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:11.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:10.702+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:11.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:10.703+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:11.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:11.574+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:11.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:11.574+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:11.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:11.574+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:11.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:11.575+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:11.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:11.687+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:11.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:11.687+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:11.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:11.687+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:11.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:11.687+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:11.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:11.694+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:11.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:11.694+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:11.785 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:11.694+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:11.785 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:11.695+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:13.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:12.728+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:13.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:12.728+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:13.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:12.728+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:13.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:12.729+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:13.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:12.674+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:13.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:12.674+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:13.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:12.674+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:13.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:12.674+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:13.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:12.607+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:13.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:12.607+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:13.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:12.607+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:13.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:12.608+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:14.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:13.715+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:14.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:13.715+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:14.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:13.715+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:14.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:13.716+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:14.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:13.633+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:14.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:13.633+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:14.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:13.633+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:14.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:13.634+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:14.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:13.605+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:14.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:13.605+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:14.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:13.605+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:14.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:13.605+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:15.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:14.584+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:15.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:14.584+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:15.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:14.584+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:15.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:14.584+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:15.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:14.744+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:15.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:14.744+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:15.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:14.744+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:15.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:14.745+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:15.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:14.601+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:15.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:14.601+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:15.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:14.601+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:15.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:14.602+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:15.752 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:07:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:07:15.390+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 16208 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:07:15.752 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:15.545+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:15.752 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:15.545+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:15.752 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:15.545+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:15.753 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:15.545+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:15.753 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:15.576+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:15.753 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:15.576+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:15.753 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:15.576+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:15.753 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:15.577+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:16.031 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:15.751+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:16.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:15.751+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:16.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:15.751+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:16.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:15.753+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:16.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:16.513+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:16.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:16.513+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:16.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:16.513+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:16.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:16.513+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:16.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:16.723+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:16.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:16.723+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:16.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:16.723+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:16.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:16.723+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:16.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:16.574+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:16.785 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:16.574+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:16.785 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:16.574+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:16.785 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:16.574+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:18.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:17.543+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:18.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:17.543+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:18.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:17.543+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:18.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:17.543+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:18.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:17.773+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:18.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:17.773+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:18.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:17.773+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:18.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:17.774+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:18.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:17.593+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:18.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:17.593+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:18.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:17.593+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:18.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:17.594+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:18.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:18.527+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:18.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:18.527+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:18.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:18.527+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:18.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:18.528+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:18.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:18.568+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:18.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:18.568+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:18.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:18.568+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:18.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:18.569+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:19.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:18.806+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:19.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:18.806+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:19.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:18.806+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:19.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:18.806+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:19.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:19.570+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:19.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:19.570+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:19.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:19.570+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:19.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:19.570+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:19.855 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:19.546+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:19.855 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:19.546+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:19.855 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:19.546+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:19.855 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:19.547+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:19.855 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:19.790+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:19.856 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:19.790+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:19.856 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:19.790+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:19.856 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:19.791+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:20.415 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:07:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:07:20.392+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 16294 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:07:20.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:20.525+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:20.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:20.525+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:20.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:20.525+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:20.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:20.527+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:20.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:20.533+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:20.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:20.533+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:20.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:20.533+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:20.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:20.533+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:21.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:20.815+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:21.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:20.815+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:21.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:20.815+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:21.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:20.816+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:21.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:21.562+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:21.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:21.562+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:21.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:21.562+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:21.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:21.563+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:21.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:21.568+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:21.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:21.568+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:21.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:21.568+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:21.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:21.569+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:21.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:21.766+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:21.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:21.766+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:21.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:21.766+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:21.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:21.767+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:22.832 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:22.817+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:22.833 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:22.817+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:22.833 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:22.817+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:22.833 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:22.817+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:22.834 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:22.619+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:22.834 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:22.619+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:22.834 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:22.619+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:22.834 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:22.619+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:22.834 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:22.575+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:22.834 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:22.575+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:22.834 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:22.575+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:22.835 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:22.576+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:23.831 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:23.536+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:23.831 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:23.536+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:23.831 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:23.536+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:23.831 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:23.536+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:23.832 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:23.607+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:23.832 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:23.607+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:23.832 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:23.607+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:23.832 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:23.608+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:24.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:23.831+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:24.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:23.831+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:24.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:23.831+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:24.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:23.832+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:24.853 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:24.544+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:24.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:24.544+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:24.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:24.544+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:24.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:24.545+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:24.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:24.641+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:24.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:24.641+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:24.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:24.641+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:24.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:24.641+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:25.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:24.853+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:25.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:24.853+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:25.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:24.853+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:25.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:24.854+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:25.668 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:07:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:07:25.394+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 16384 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:07:25.669 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:25.525+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:25.669 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:25.525+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:25.669 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:25.525+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:25.669 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:25.526+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:26.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:25.875+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:26.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:25.875+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:26.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:25.875+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:26.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:25.875+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:26.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:25.668+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:26.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:25.668+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:26.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:25.668+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:26.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:25.669+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:26.835 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:26.714+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:26.836 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:26.714+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:26.836 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:26.714+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:26.836 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:26.715+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:26.836 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:26.549+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:26.836 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:26.549+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:26.837 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:26.549+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:26.837 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:26.550+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:27.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:26.835+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:27.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:26.835+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:27.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:26.835+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:27.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:26.836+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:27.821 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:27.739+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:27.822 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:27.739+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:27.822 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:27.739+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:27.822 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:27.740+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:27.822 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:27.569+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:27.822 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:27.569+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:27.823 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:27.569+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:27.823 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:27.570+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:28.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:27.821+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:28.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:27.821+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:28.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:27.821+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:28.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:27.822+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:28.978 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:28.755+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:28.979 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:28.755+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:28.979 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:28.755+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:28.979 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:28.757+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:28.980 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:28.573+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:28.980 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:28.573+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:28.980 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:28.573+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:28.980 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:28.574+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:28.980 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:28.803+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:28.981 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:28.803+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:28.981 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:28.803+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:28.981 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:28.803+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:29.848 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:29.782+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:29.848 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:29.782+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:29.849 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:29.782+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:29.849 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:29.783+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:29.849 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:29.562+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:29.849 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:29.562+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:29.849 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:29.562+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:29.850 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:29.563+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:30.161 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:29.848+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:30.161 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:29.848+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:30.161 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:29.848+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:30.161 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:29.849+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:30.417 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:07:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:07:30.397+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 16469 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:07:30.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:30.577+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:30.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:30.577+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:30.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:30.577+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:30.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:30.577+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:31.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:30.812+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:31.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:30.812+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:31.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:30.812+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:31.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:30.812+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:31.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:30.882+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:31.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:30.882+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:31.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:30.882+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:31.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:30.882+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:31.843 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:31.831+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:31.844 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:31.831+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:31.844 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:31.831+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:31.844 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:31.832+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:31.844 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:31.592+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:31.844 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:31.592+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:31.845 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:31.592+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:31.845 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:31.593+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:32.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:31.844+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:32.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:31.844+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:32.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:31.844+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:32.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:31.845+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:33.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:32.861+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:33.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:32.861+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:33.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:32.861+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:33.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:32.862+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:33.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:32.622+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:33.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:32.622+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:33.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:32.622+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:33.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:32.623+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:33.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:32.865+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:33.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:32.865+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:33.035 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:32.865+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:33.035 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:32.865+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:33.853 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:33.617+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:33.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:33.617+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:33.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:33.617+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:33.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:33.618+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:34.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:33.854+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:34.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:33.854+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:34.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:33.854+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:34.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:33.855+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:34.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:33.911+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:34.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:33.911+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:34.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:33.911+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:34.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:33.911+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:34.951 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:34.879+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:34.952 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:34.879+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:34.952 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:34.879+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:34.952 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:34.879+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:34.952 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:34.654+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:34.953 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:34.654+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:34.953 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:34.654+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:34.953 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:34.655+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:35.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:34.952+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:35.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:34.952+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:35.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:34.952+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:35.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:34.953+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:35.649 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:07:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:07:35.399+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 16558 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:07:35.903 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:35.649+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:35.903 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:35.649+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:35.903 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:35.649+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:35.903 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:35.651+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:36.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:35.903+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:36.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:35.903+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:36.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:35.903+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:36.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:35.904+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:36.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:35.926+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:36.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:35.926+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:36.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:35.926+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:36.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:35.927+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:36.944 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:36.642+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:36.944 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:36.642+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:36.944 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:36.642+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:36.944 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:36.643+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:37.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:36.945+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:37.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:36.945+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:37.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:36.945+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:37.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:36.946+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:37.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:36.945+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:37.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:36.945+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:37.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:36.945+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:37.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:36.946+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:37.929 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:37.658+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:37.929 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:37.658+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:37.929 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:37.658+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:37.929 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:37.659+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:38.193 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:37.967+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:38.193 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:37.967+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:38.193 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:37.967+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:38.193 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:37.967+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:38.193 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:37.929+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:38.194 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:37.929+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:38.194 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:37.929+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:38.194 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:37.930+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:38.947 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:38.684+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:38.947 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:38.684+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:38.947 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:38.684+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:38.948 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:38.685+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:38.948 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:38.894+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:38.948 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:38.894+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:38.948 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:38.894+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:38.948 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:38.895+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:39.281 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:38.948+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:39.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:38.948+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:39.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:38.948+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:39.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:38.949+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:40.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:39.907+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:40.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:39.907+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:40.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:39.907+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:40.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:39.908+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:40.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:39.921+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:40.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:39.921+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:40.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:39.921+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:40.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:39.922+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:40.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:39.675+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:40.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:39.675+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:40.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:39.675+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:40.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:39.676+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:40.412 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:07:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:07:40.400+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 16644 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:07:41.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:40.696+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:41.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:40.696+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:41.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:40.696+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:41.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:40.697+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:41.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:40.902+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:41.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:40.902+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:41.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:40.902+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:41.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:40.903+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:41.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:40.869+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:41.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:40.869+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:41.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:40.869+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:41.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:40.870+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:42.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:41.727+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:42.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:41.727+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:42.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:41.727+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:42.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:41.727+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:42.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:41.886+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:42.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:41.886+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:42.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:41.886+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:42.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:41.887+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:42.035 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:41.876+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:42.035 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:41.876+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:42.035 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:41.876+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:42.036 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:41.876+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:43.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:42.742+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:43.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:42.742+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:43.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:42.742+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:43.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:42.742+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:43.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:42.839+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:43.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:42.839+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:43.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:42.839+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:43.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:42.839+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:43.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:42.878+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:43.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:42.878+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:43.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:42.878+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:43.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:42.879+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:43.919 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:43.748+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:43.919 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:43.748+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:43.919 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:43.748+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:43.919 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:43.749+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:43.920 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:43.815+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:43.920 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:43.815+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:43.920 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:43.815+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:43.920 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:43.816+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:44.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:43.918+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:44.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:43.918+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:44.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:43.918+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:44.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:43.919+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:45.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:44.783+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:45.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:44.783+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:45.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:44.783+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:45.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:44.783+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:45.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:44.939+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:45.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:44.939+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:45.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:44.939+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:45.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:44.940+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:45.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:44.814+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:45.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:44.814+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:45.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:44.814+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:45.285 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:44.814+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:45.765 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:07:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:07:45.401+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 16734 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:07:46.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:45.765+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:46.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:45.765+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:46.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:45.765+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:46.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:45.766+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:46.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:45.930+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:46.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:45.930+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:46.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:45.930+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:46.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:45.931+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:46.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:45.812+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:46.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:45.812+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:46.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:45.812+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:46.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:45.813+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:47.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:46.742+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:47.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:46.742+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:47.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:46.742+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:47.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:46.743+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:47.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:46.946+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:47.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:46.946+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:47.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:46.946+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:47.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:46.946+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:47.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:46.829+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:47.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:46.829+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:47.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:46.829+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:47.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:46.830+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:48.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:47.836+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:48.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:47.836+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:48.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:47.836+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:48.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:47.837+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:48.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:47.897+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:48.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:47.897+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:48.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:47.897+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:48.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:47.898+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:48.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:47.722+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:48.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:47.722+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:48.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:47.722+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:48.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:47.722+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:49.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:48.924+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:49.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:48.924+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:49.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:48.924+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:49.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:48.925+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:49.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:48.794+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:49.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:48.794+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:49.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:48.794+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:49.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:48.795+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:49.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:48.704+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:49.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:48.704+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:49.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:48.704+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:49.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:48.706+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:50.035 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:49.720+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:50.035 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:49.720+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:50.035 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:49.720+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:50.036 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:49.721+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:50.036 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:49.780+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:50.036 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:49.780+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:50.036 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:49.780+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:50.036 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:49.781+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:50.037 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:49.940+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:50.037 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:49.940+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:50.037 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:49.940+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:50.037 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:49.941+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:50.413 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:07:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:07:50.403+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 16819 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:07:51.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:50.753+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:51.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:50.753+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:51.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:50.753+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:51.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:50.754+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:51.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:50.917+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:51.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:50.917+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:51.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:50.917+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:51.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:50.918+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:51.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:50.730+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:51.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:50.730+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:51.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:50.730+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:51.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:50.731+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:52.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:51.869+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:52.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:51.869+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:52.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:51.869+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:52.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:51.870+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:52.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:51.780+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:52.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:51.780+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:52.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:51.780+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:52.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:51.781+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:52.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:51.750+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:52.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:51.750+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:52.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:51.750+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:52.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:51.750+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:53.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:52.820+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:53.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:52.820+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:53.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:52.820+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:53.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:52.821+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:53.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:52.735+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:53.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:52.735+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:53.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:52.735+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:53.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:52.736+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:53.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:52.865+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:53.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:52.865+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:53.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:52.865+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:53.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:52.866+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:53.844 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:53.775+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:53.844 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:53.775+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:53.844 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:53.775+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:53.845 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:53.776+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:53.845 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:53.785+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:53.845 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:53.785+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:53.846 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:53.785+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:53.846 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:53.785+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:54.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:53.843+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:54.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:53.843+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:54.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:53.843+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:54.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:53.844+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:55.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:54.873+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:55.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:54.873+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:55.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:54.873+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:55.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:54.873+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:55.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:54.784+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:55.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:54.784+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:55.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:54.784+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:55.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:54.785+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:55.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:54.756+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:55.035 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:54.756+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:55.035 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:54.756+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:55.035 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:54.756+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:55.724 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:07:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:07:55.405+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 16907 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:07:56.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:55.901+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:56.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:55.901+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:56.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:55.901+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:56.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:55.902+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:56.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:55.724+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:56.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:55.724+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:56.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:55.724+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:56.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:55.725+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:56.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:55.750+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:56.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:55.751+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:56.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:55.751+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:56.035 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:55.752+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:57.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:56.717+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:57.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:56.717+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:57.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:56.717+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:57.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:56.718+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:57.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:56.682+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:57.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:56.682+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:57.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:56.682+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:57.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:56.683+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:57.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:56.866+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:57.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:56.866+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:57.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:56.866+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:57.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:56.867+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:58.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:57.732+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:58.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:57.732+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:58.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:57.732+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:58.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:57.732+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:58.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:57.693+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:58.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:57.693+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:58.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:57.693+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:58.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:57.693+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:58.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:57.870+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:58.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:57.870+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:58.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:57.870+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:58.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:57.871+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:59.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:58.777+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:59.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:58.777+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:59.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:58.777+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:07:59.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:58.778+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:59.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:58.680+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:59.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:58.680+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:59.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:58.680+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:07:59.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:58.682+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:07:59.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:58.918+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:59.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:58.918+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:59.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:58.919+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:07:59.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:58.919+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:00.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:59.726+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:00.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:59.726+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:00.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:59.726+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:00.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:07:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:07:59.728+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:00.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:59.821+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:00.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:59.821+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:00.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:59.821+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:00.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:07:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:07:59.822+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:00.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:59.875+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:00.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:59.875+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:00.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:59.875+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:00.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:07:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:07:59.876+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:00.419 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:08:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:08:00.407+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 16993 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:08:01.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:00.798+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:01.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:00.798+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:01.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:00.798+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:01.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:00.799+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:01.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:00.708+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:01.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:00.708+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:01.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:00.708+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:01.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:00.709+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:01.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:00.838+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:01.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:00.838+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:01.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:00.838+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:01.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:00.838+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:02.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:01.793+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:02.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:01.793+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:02.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:01.793+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:02.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:01.793+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:02.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:01.729+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:02.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:01.729+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:02.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:01.729+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:02.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:01.729+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:02.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:01.847+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:02.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:01.847+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:02.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:01.847+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:02.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:01.848+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:02.959 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-24T10:08:02.960 DEBUG:tasks.cephfs.mount:Force/lazy unmounting on client.0 2024-03-24T10:08:02.960 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:08:02.960 DEBUG:teuthology.orchestra.run.smithi022:> sudo umount --lazy --force /home/ubuntu/cephtest/mnt.0 2024-03-24T10:08:02.984 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:02.803+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:02.985 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:02.803+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:02.985 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:02.803+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:02.985 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:02.804+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:02.985 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:02.799+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:02.985 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:02.799+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:02.985 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:02.799+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:02.985 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:02.800+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:02.986 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:02.728+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:02.986 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:02.728+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:02.986 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:02.728+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:02.986 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:02.728+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:02.997 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/vfork/vfork02.c: Cannot open: Input/output error 2024-03-24T10:08:02.997 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/vfork/Makefile: Cannot open: Input/output error 2024-03-24T10:08:02.997 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/vfork/vfork01.c: Cannot open: Input/output error 2024-03-24T10:08:02.997 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/vfork: Cannot utime: Input/output error 2024-03-24T10:08:02.997 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/gettid: Cannot mkdir: Input/output error 2024-03-24T10:08:02.997 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/gettid/Makefile: Cannot open: Input/output error 2024-03-24T10:08:02.998 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/gettid/gettid01.c: Cannot open: Input/output error 2024-03-24T10:08:02.998 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mallopt: Cannot mkdir: Input/output error 2024-03-24T10:08:02.998 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mallopt/Makefile: Cannot open: Input/output error 2024-03-24T10:08:02.998 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mallopt/mallopt01.c: Cannot open: Input/output error 2024-03-24T10:08:02.998 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/execle: Cannot mkdir: Input/output error 2024-03-24T10:08:02.998 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/execle/Makefile: Cannot open: Input/output error 2024-03-24T10:08:02.998 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/execle/execle01.c: Cannot open: Input/output error 2024-03-24T10:08:02.998 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sgetmask: Cannot mkdir: Input/output error 2024-03-24T10:08:02.999 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sgetmask/sgetmask01.c: Cannot open: Input/output error 2024-03-24T10:08:02.999 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sgetmask/Makefile: Cannot open: Input/output error 2024-03-24T10:08:02.999 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/llseek: Cannot mkdir: Input/output error 2024-03-24T10:08:02.999 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/llseek/llseek01.c: Cannot open: Input/output error 2024-03-24T10:08:02.999 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/llseek/Makefile: Cannot open: Input/output error 2024-03-24T10:08:02.999 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/llseek/llseek02.c: Cannot open: Input/output error 2024-03-24T10:08:02.999 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setfsuid: Cannot mkdir: Input/output error 2024-03-24T10:08:02.999 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setfsuid/setfsuid02.c: Cannot open: Input/output error 2024-03-24T10:08:02.999 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setfsuid/setfsuid01.c: Cannot open: Input/output error 2024-03-24T10:08:02.999 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setfsuid/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.000 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setfsuid/setfsuid03.c: Cannot open: Input/output error 2024-03-24T10:08:03.000 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setfsuid/setfsuid04.c: Cannot open: Input/output error 2024-03-24T10:08:03.000 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mq_notify: Cannot mkdir: Input/output error 2024-03-24T10:08:03.000 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mq_notify/mq_notify01.c: Cannot open: Input/output error 2024-03-24T10:08:03.000 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mq_notify/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.000 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_yield: Cannot mkdir: Input/output error 2024-03-24T10:08:03.000 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_yield/sched_yield01.c: Cannot open: Input/output error 2024-03-24T10:08:03.000 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_yield/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.000 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/pathconf: Cannot mkdir: Input/output error 2024-03-24T10:08:03.001 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/pathconf/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.001 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/pathconf/pathconf01.c: Cannot open: Input/output error 2024-03-24T10:08:03.001 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/keyctl: Cannot mkdir: Input/output error 2024-03-24T10:08:03.001 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/keyctl/keyctl01.c: Cannot open: Input/output error 2024-03-24T10:08:03.001 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/keyctl/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.001 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fchmod: Cannot mkdir: Input/output error 2024-03-24T10:08:03.001 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fchmod/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.001 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fchmod/fchmod07.c: Cannot open: Input/output error 2024-03-24T10:08:03.001 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fchmod/fchmod06.c: Cannot open: Input/output error 2024-03-24T10:08:03.002 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fchmod/change_owner.mode.sh: Cannot open: Input/output error 2024-03-24T10:08:03.002 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fchmod/fchmod04.c: Cannot open: Input/output error 2024-03-24T10:08:03.002 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fchmod/change_owner.c: Cannot open: Input/output error 2024-03-24T10:08:03.002 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fchmod/fchmod05.c: Cannot open: Input/output error 2024-03-24T10:08:03.002 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fchmod/fchmod02.c: Cannot open: Input/output error 2024-03-24T10:08:03.002 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fchmod/fchmod03.c: Cannot open: Input/output error 2024-03-24T10:08:03.002 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fchmod/fchmod01.c: Cannot open: Input/output error 2024-03-24T10:08:03.002 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/accept: Cannot mkdir: Input/output error 2024-03-24T10:08:03.002 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/accept/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.002 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/accept/accept01.c: Cannot open: Input/output error 2024-03-24T10:08:03.003 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/abort: Cannot mkdir: Input/output error 2024-03-24T10:08:03.003 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/abort/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.003 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/abort/abort01.c: Cannot open: Input/output error 2024-03-24T10:08:03.003 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getgroups: Cannot mkdir: Input/output error 2024-03-24T10:08:03.003 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getgroups/getgroups04.c: Cannot open: Input/output error 2024-03-24T10:08:03.003 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getgroups/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.003 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getgroups/getgroups02.c: Cannot open: Input/output error 2024-03-24T10:08:03.004 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getgroups/getgroups01.c: Cannot open: Input/output error 2024-03-24T10:08:03.004 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getgroups/getgroups03.c: Cannot open: Input/output error 2024-03-24T10:08:03.004 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fchmodat: Cannot mkdir: Input/output error 2024-03-24T10:08:03.004 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fchmodat/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.018 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fchmodat/fchmodat01.c: Cannot open: Input/output error 2024-03-24T10:08:03.018 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/io_setup: Cannot mkdir: Input/output error 2024-03-24T10:08:03.018 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/io_setup/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.018 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/io_setup/io_setup01.c: Cannot open: Input/output error 2024-03-24T10:08:03.018 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/umask: Cannot mkdir: Input/output error 2024-03-24T10:08:03.018 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/umask/umask01.c: Cannot open: Input/output error 2024-03-24T10:08:03.018 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/umask/umask03.c: Cannot open: Input/output error 2024-03-24T10:08:03.019 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/umask/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.019 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/umask/umask02.c: Cannot open: Input/output error 2024-03-24T10:08:03.019 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mknodat: Cannot mkdir: Input/output error 2024-03-24T10:08:03.019 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mknodat/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.019 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mknodat/mknodat01.c: Cannot open: Input/output error 2024-03-24T10:08:03.019 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getsockname: Cannot mkdir: Input/output error 2024-03-24T10:08:03.019 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getsockname/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.019 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getsockname/getsockname01.c: Cannot open: Input/output error 2024-03-24T10:08:03.019 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/splice: Cannot mkdir: Input/output error 2024-03-24T10:08:03.020 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/splice/splice01.c: Cannot open: Input/output error 2024-03-24T10:08:03.020 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/splice/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.020 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/splice/splice02.c: Cannot open: Input/output error 2024-03-24T10:08:03.020 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setgid: Cannot mkdir: Input/output error 2024-03-24T10:08:03.020 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setgid/compat_16.h: Cannot open: Input/output error 2024-03-24T10:08:03.020 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setgid/setgid01.c: Cannot open: Input/output error 2024-03-24T10:08:03.020 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setgid/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.020 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setgid/setgid02.c: Cannot open: Input/output error 2024-03-24T10:08:03.020 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setgid/setgid03.c: Cannot open: Input/output error 2024-03-24T10:08:03.020 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/dup3: Cannot mkdir: Input/output error 2024-03-24T10:08:03.021 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/dup3/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.021 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/dup3/dup3_01.c: Cannot open: Input/output error 2024-03-24T10:08:03.021 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setregid: Cannot mkdir: Input/output error 2024-03-24T10:08:03.021 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setregid/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.021 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setregid/setregid02.c: Cannot open: Input/output error 2024-03-24T10:08:03.021 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setregid/setregid03.c: Cannot open: Input/output error 2024-03-24T10:08:03.021 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setregid/setregid01.c: Cannot open: Input/output error 2024-03-24T10:08:03.021 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setregid/setregid04.c: Cannot open: Input/output error 2024-03-24T10:08:03.021 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/prctl: Cannot mkdir: Input/output error 2024-03-24T10:08:03.021 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/prctl/prctl02.c: Cannot open: Input/output error 2024-03-24T10:08:03.022 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/prctl/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.022 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/prctl/prctl01.c: Cannot open: Input/output error 2024-03-24T10:08:03.022 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigwait: Cannot mkdir: Input/output error 2024-03-24T10:08:03.022 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigwait/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.022 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/exit_group: Cannot mkdir: Input/output error 2024-03-24T10:08:03.022 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/exit_group/exit_group01.c: Cannot open: Input/output error 2024-03-24T10:08:03.022 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/exit_group/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.022 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fdatasync: Cannot mkdir: Input/output error 2024-03-24T10:08:03.023 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fdatasync/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.023 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fdatasync/fdatasync02.c: Cannot open: Input/output error 2024-03-24T10:08:03.023 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fdatasync/fdatasync01.c: Cannot open: Input/output error 2024-03-24T10:08:03.023 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/personality: Cannot mkdir: Input/output error 2024-03-24T10:08:03.023 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/personality/personality02.c: Cannot open: Input/output error 2024-03-24T10:08:03.023 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/personality/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.023 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/personality/personality01.c: Cannot open: Input/output error 2024-03-24T10:08:03.023 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/pselect: Cannot mkdir: Input/output error 2024-03-24T10:08:03.023 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/pselect/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.023 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/pselect/pselect01.c: Cannot open: Input/output error 2024-03-24T10:08:03.024 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mount: Cannot mkdir: Input/output error 2024-03-24T10:08:03.024 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mount/mount01.c: Cannot open: Input/output error 2024-03-24T10:08:03.024 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mount/setuid_test.c: Cannot open: Input/output error 2024-03-24T10:08:03.024 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mount/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.024 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mount/mount03.c: Cannot open: Input/output error 2024-03-24T10:08:03.024 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mount/mount04.c: Cannot open: Input/output error 2024-03-24T10:08:03.024 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mount/test_mount: Cannot open: Input/output error 2024-03-24T10:08:03.024 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mount/mount02.c: Cannot open: Input/output error 2024-03-24T10:08:03.024 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_get_priority_max: Cannot mkdir: Input/output error 2024-03-24T10:08:03.025 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_get_priority_max/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.025 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_get_priority_max/sched_get_priority_max02.c: Cannot open: Input/output error 2024-03-24T10:08:03.025 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_get_priority_max/sched_get_priority_max01.c: Cannot open: Input/output error 2024-03-24T10:08:03.025 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rt_sigsuspend: Cannot mkdir: Input/output error 2024-03-24T10:08:03.025 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rt_sigsuspend/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.025 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rt_sigsuspend/rt_sigsuspend01.c: Cannot open: Input/output error 2024-03-24T10:08:03.025 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setresuid: Cannot mkdir: Input/output error 2024-03-24T10:08:03.025 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setresuid/setresuid04.c: Cannot open: Input/output error 2024-03-24T10:08:03.025 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setresuid/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.026 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setresuid/setresuid02.c: Cannot open: Input/output error 2024-03-24T10:08:03.026 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setresuid/setresuid01.c: Cannot open: Input/output error 2024-03-24T10:08:03.026 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setresuid/setresuid03.c: Cannot open: Input/output error 2024-03-24T10:08:03.026 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_setparam: Cannot mkdir: Input/output error 2024-03-24T10:08:03.026 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_setparam/sched_setparam05.c: Cannot open: Input/output error 2024-03-24T10:08:03.026 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_setparam/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.026 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_setparam/sched_setparam04.c: Cannot open: Input/output error 2024-03-24T10:08:03.026 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_setparam/sched_setparam03.c: Cannot open: Input/output error 2024-03-24T10:08:03.027 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_setparam/sched_setparam02.c: Cannot open: Input/output error 2024-03-24T10:08:03.027 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_setparam/sched_setparam01.c: Cannot open: Input/output error 2024-03-24T10:08:03.027 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/recvmsg: Cannot mkdir: Input/output error 2024-03-24T10:08:03.027 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/recvmsg/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.027 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/recvmsg/recvmsg01.c: Cannot open: Input/output error 2024-03-24T10:08:03.027 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/ulimit: Cannot mkdir: Input/output error 2024-03-24T10:08:03.027 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/ulimit/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.027 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/ulimit/README: Cannot open: Input/output error 2024-03-24T10:08:03.027 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/ulimit/ulimit01.c: Cannot open: Input/output error 2024-03-24T10:08:03.027 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.028 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setfsgid: Cannot mkdir: Input/output error 2024-03-24T10:08:03.028 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setfsgid/setfsgid02.c: Cannot open: Input/output error 2024-03-24T10:08:03.028 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setfsgid/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.028 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setfsgid/setfsgid03.c: Cannot open: Input/output error 2024-03-24T10:08:03.028 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setfsgid/setfsgid01.c: Cannot open: Input/output error 2024-03-24T10:08:03.028 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/unlink: Cannot mkdir: Input/output error 2024-03-24T10:08:03.028 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/unlink/unlink05.c: Cannot open: Input/output error 2024-03-24T10:08:03.028 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/unlink/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.028 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/unlink/unlink07.c: Cannot open: Input/output error 2024-03-24T10:08:03.028 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/unlink/unlink08.c: Cannot open: Input/output error 2024-03-24T10:08:03.029 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/unlink/unlink06.c: Cannot open: Input/output error 2024-03-24T10:08:03.029 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setpgrp: Cannot mkdir: Input/output error 2024-03-24T10:08:03.029 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setpgrp/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.029 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setpgrp/setpgrp02.c: Cannot open: Input/output error 2024-03-24T10:08:03.029 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setpgrp/setpgrp01.c: Cannot open: Input/output error 2024-03-24T10:08:03.029 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/waitpid: Cannot mkdir: Input/output error 2024-03-24T10:08:03.029 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/waitpid/waitpid04.c: Cannot open: Input/output error 2024-03-24T10:08:03.029 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/waitpid/waitpid10.c: Cannot open: Input/output error 2024-03-24T10:08:03.029 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/waitpid/waitpid09.c: Cannot open: Input/output error 2024-03-24T10:08:03.029 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/waitpid/waitpid07.c: Cannot open: Input/output error 2024-03-24T10:08:03.030 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/waitpid/waitpid02.c: Cannot open: Input/output error 2024-03-24T10:08:03.030 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/waitpid/waitpid01.c: Cannot open: Input/output error 2024-03-24T10:08:03.030 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/waitpid/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.030 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/waitpid/waitpid13.c: Cannot open: Input/output error 2024-03-24T10:08:03.030 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/waitpid/waitpid05.c: Cannot open: Input/output error 2024-03-24T10:08:03.030 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/waitpid/waitpid08.c: Cannot open: Input/output error 2024-03-24T10:08:03.030 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/waitpid/waitpid11.c: Cannot open: Input/output error 2024-03-24T10:08:03.030 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/waitpid/waitpid03.c: Cannot open: Input/output error 2024-03-24T10:08:03.030 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/waitpid/waitpid12.c: Cannot open: Input/output error 2024-03-24T10:08:03.031 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/waitpid/waitpid06.c: Cannot open: Input/output error 2024-03-24T10:08:03.031 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/io_destroy: Cannot mkdir: Input/output error 2024-03-24T10:08:03.031 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/io_destroy/io_destroy01.c: Cannot open: Input/output error 2024-03-24T10:08:03.031 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/io_destroy/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.031 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/ustat: Cannot mkdir: Input/output error 2024-03-24T10:08:03.031 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/ustat/ustat02.c: Cannot open: Input/output error 2024-03-24T10:08:03.031 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/ustat/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.031 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/ustat/ustat01.c: Cannot open: Input/output error 2024-03-24T10:08:03.031 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigsuspend: Cannot mkdir: Input/output error 2024-03-24T10:08:03.032 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigsuspend/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.032 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigsuspend/sigsuspend01.c: Cannot open: Input/output error 2024-03-24T10:08:03.032 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mq_unlink: Cannot mkdir: Input/output error 2024-03-24T10:08:03.032 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mq_unlink/mq_unlink01.c: Cannot open: Input/output error 2024-03-24T10:08:03.032 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mq_unlink/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.032 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getcpu: Cannot mkdir: Input/output error 2024-03-24T10:08:03.032 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getcpu/getcpu01.c: Cannot open: Input/output error 2024-03-24T10:08:03.032 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getcpu/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.032 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/pipe2: Cannot mkdir: Input/output error 2024-03-24T10:08:03.032 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/pipe2/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.033 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/pipe2/pipe2_01.c: Cannot open: Input/output error 2024-03-24T10:08:03.033 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/pipe2/pipe2_02.c: Cannot open: Input/output error 2024-03-24T10:08:03.033 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mkdir: Cannot mkdir: Input/output error 2024-03-24T10:08:03.033 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mkdir/mkdir09.c: Cannot open: Input/output error 2024-03-24T10:08:03.033 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mkdir/mkdir08.c: Cannot open: Input/output error 2024-03-24T10:08:03.033 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mkdir/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.033 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mkdir/mkdir01.c: Cannot open: Input/output error 2024-03-24T10:08:03.033 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mkdir/mkdir04.c: Cannot open: Input/output error 2024-03-24T10:08:03.033 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mkdir/mkdir05.c: Cannot open: Input/output error 2024-03-24T10:08:03.034 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mkdir/mkdir02.c: Cannot open: Input/output error 2024-03-24T10:08:03.034 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mkdir/mkdir03.c: Cannot open: Input/output error 2024-03-24T10:08:03.034 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/uname: Cannot mkdir: Input/output error 2024-03-24T10:08:03.034 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/uname/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.034 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/uname/uname01.c: Cannot open: Input/output error 2024-03-24T10:08:03.034 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/uname/uname03.c: Cannot open: Input/output error 2024-03-24T10:08:03.034 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/uname/uname02.c: Cannot open: Input/output error 2024-03-24T10:08:03.034 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getegid: Cannot mkdir: Input/output error 2024-03-24T10:08:03.034 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getegid/getegid02.c: Cannot open: Input/output error 2024-03-24T10:08:03.035 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getegid/compat_16.h: Cannot open: Input/output error 2024-03-24T10:08:03.035 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getegid/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.035 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getegid/getegid01.c: Cannot open: Input/output error 2024-03-24T10:08:03.035 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/read: Cannot mkdir: Input/output error 2024-03-24T10:08:03.035 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/read/read03.c: Cannot open: Input/output error 2024-03-24T10:08:03.035 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/read/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.035 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/read/read02.c: Cannot open: Input/output error 2024-03-24T10:08:03.035 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/read/read04.c: Cannot open: Input/output error 2024-03-24T10:08:03.035 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/read/read01.c: Cannot open: Input/output error 2024-03-24T10:08:03.035 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mlock: Cannot mkdir: Input/output error 2024-03-24T10:08:03.036 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mlock/mlock01.c: Cannot open: Input/output error 2024-03-24T10:08:03.036 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mlock/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.036 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mlock/mlock02.c: Cannot open: Input/output error 2024-03-24T10:08:03.036 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/poll: Cannot mkdir: Input/output error 2024-03-24T10:08:03.036 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/poll/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.036 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/poll/poll01.c: Cannot open: Input/output error 2024-03-24T10:08:03.036 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/io_cancel: Cannot mkdir: Input/output error 2024-03-24T10:08:03.036 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/io_cancel/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.036 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/io_cancel/io_cancel01.c: Cannot open: Input/output error 2024-03-24T10:08:03.037 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/execlp: Cannot mkdir: Input/output error 2024-03-24T10:08:03.037 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/execlp/execlp01.c: Cannot open: Input/output error 2024-03-24T10:08:03.037 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/execlp/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.037 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/stat: Cannot mkdir: Input/output error 2024-03-24T10:08:03.037 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/stat/stat06.c: Cannot open: Input/output error 2024-03-24T10:08:03.037 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/stat/stat02.c: Cannot open: Input/output error 2024-03-24T10:08:03.037 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/stat/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.037 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/stat/stat03.c: Cannot open: Input/output error 2024-03-24T10:08:03.037 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/stat/stat05.c: Cannot open: Input/output error 2024-03-24T10:08:03.037 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/stat/stat01.c: Cannot open: Input/output error 2024-03-24T10:08:03.038 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fstatfs: Cannot mkdir: Input/output error 2024-03-24T10:08:03.038 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fstatfs/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.038 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fstatfs/fstatfs01.c: Cannot open: Input/output error 2024-03-24T10:08:03.038 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fstatfs/fstatfs02.c: Cannot open: Input/output error 2024-03-24T10:08:03.038 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/clock_nanosleep: Cannot mkdir: Input/output error 2024-03-24T10:08:03.038 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/clock_nanosleep/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.038 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/clock_nanosleep/clock_nanosleep01.c: Cannot open: Input/output error 2024-03-24T10:08:03.038 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/symlink: Cannot mkdir: Input/output error 2024-03-24T10:08:03.038 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/symlink/symlink02.c: Cannot open: Input/output error 2024-03-24T10:08:03.039 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/symlink/symlink01.c: Cannot open: Input/output error 2024-03-24T10:08:03.039 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/symlink/symlink05.c: Cannot open: Input/output error 2024-03-24T10:08:03.039 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/symlink/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.039 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/symlink/symlink03.c: Cannot open: Input/output error 2024-03-24T10:08:03.039 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/symlink/symlink04.c: Cannot open: Input/output error 2024-03-24T10:08:03.039 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mprotect: Cannot mkdir: Input/output error 2024-03-24T10:08:03.039 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mprotect/mprotect01.c: Cannot open: Input/output error 2024-03-24T10:08:03.039 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mprotect/mprotect02.c: Cannot open: Input/output error 2024-03-24T10:08:03.039 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mprotect/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.039 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mprotect/mprotect03.c: Cannot open: Input/output error 2024-03-24T10:08:03.040 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rmdir: Cannot mkdir: Input/output error 2024-03-24T10:08:03.040 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rmdir/rmdir02.c: Cannot open: Input/output error 2024-03-24T10:08:03.040 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rmdir/rmdir04.c: Cannot open: Input/output error 2024-03-24T10:08:03.040 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rmdir/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.040 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rmdir/rmdir03.c: Cannot open: Input/output error 2024-03-24T10:08:03.040 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rmdir/rmdir05.c: Cannot open: Input/output error 2024-03-24T10:08:03.040 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rmdir/rmdir01.c: Cannot open: Input/output error 2024-03-24T10:08:03.040 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigaction: Cannot mkdir: Input/output error 2024-03-24T10:08:03.040 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigaction/sigaction01.c: Cannot open: Input/output error 2024-03-24T10:08:03.040 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigaction/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.041 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigaction/sigaction02.c: Cannot open: Input/output error 2024-03-24T10:08:03.041 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw: Cannot mkdir: Input/output error 2024-03-24T10:08:03.041 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw/lib.c: Cannot open: Input/output error 2024-03-24T10:08:03.041 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw/nftw.h: Cannot open: Input/output error 2024-03-24T10:08:03.041 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw/lib64.c: Cannot open: Input/output error 2024-03-24T10:08:03.041 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw/test.c: Cannot open: Input/output error 2024-03-24T10:08:03.041 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw/test_func.c: Cannot open: Input/output error 2024-03-24T10:08:03.041 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw/test_func64.c: Cannot open: Input/output error 2024-03-24T10:08:03.041 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.042 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw/nftw64.c: Cannot open: Input/output error 2024-03-24T10:08:03.042 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi022.front.sepia.ceph.com 2024-03-24T10:08:03.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T10:08:03.043 DEBUG:teuthology.orchestra.run.smithi022:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T10:08:03.049 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-03-24T10:08:03.049 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw/test64.c: Cannot open: Input/output error 2024-03-24T10:08:03.049 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw/nftw64.h: Cannot open: Input/output error 2024-03-24T10:08:03.049 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw/tools.c: Cannot open: Input/output error 2024-03-24T10:08:03.049 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw/nftw.c: Cannot open: Input/output error 2024-03-24T10:08:03.049 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw/tools64.c: Cannot open: Input/output error 2024-03-24T10:08:03.050 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mq_timedsend: Cannot mkdir: Input/output error 2024-03-24T10:08:03.050 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mq_timedsend/mq_timedsend01.c: Cannot open: Input/output error 2024-03-24T10:08:03.050 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mq_timedsend/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.050 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/statvfs: Cannot mkdir: Input/output error 2024-03-24T10:08:03.050 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/statvfs/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.050 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/statvfs/statvfs01.c: Cannot open: Input/output error 2024-03-24T10:08:03.050 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/reboot: Cannot mkdir: Input/output error 2024-03-24T10:08:03.050 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/reboot/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.050 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/reboot/reboot01.c: Cannot open: Input/output error 2024-03-24T10:08:03.050 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/reboot/reboot02.c: Cannot open: Input/output error 2024-03-24T10:08:03.051 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/confstr: Cannot mkdir: Input/output error 2024-03-24T10:08:03.051 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/confstr/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.051 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/confstr/confstr01.c: Cannot open: Input/output error 2024-03-24T10:08:03.051 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sysinfo: Cannot mkdir: Input/output error 2024-03-24T10:08:03.051 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sysinfo/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.051 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sysinfo/sysinfo02.c: Cannot open: Input/output error 2024-03-24T10:08:03.051 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sysinfo/sysinfo01.c: Cannot open: Input/output error 2024-03-24T10:08:03.051 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/execv: Cannot mkdir: Input/output error 2024-03-24T10:08:03.051 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/execv/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.052 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/execv/execv01.c: Cannot open: Input/output error 2024-03-24T10:08:03.052 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/cacheflush: Cannot mkdir: Input/output error 2024-03-24T10:08:03.052 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/cacheflush/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.052 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/cacheflush/cacheflush01.c: Cannot open: Input/output error 2024-03-24T10:08:03.052 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nanosleep: Cannot mkdir: Input/output error 2024-03-24T10:08:03.052 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nanosleep/nanosleep01.c: Cannot open: Input/output error 2024-03-24T10:08:03.052 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nanosleep/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.052 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nanosleep/nanosleep04.c: Cannot open: Input/output error 2024-03-24T10:08:03.052 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nanosleep/nanosleep03.c: Cannot open: Input/output error 2024-03-24T10:08:03.052 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nanosleep/nanosleep02.c: Cannot open: Input/output error 2024-03-24T10:08:03.053 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mq_open: Cannot mkdir: Input/output error 2024-03-24T10:08:03.053 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mq_open/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.053 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mq_open/mq_open01.c: Cannot open: Input/output error 2024-03-24T10:08:03.053 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getgid: Cannot mkdir: Input/output error 2024-03-24T10:08:03.053 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getgid/compat_16.h: Cannot open: Input/output error 2024-03-24T10:08:03.053 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getgid/getgid03.c: Cannot open: Input/output error 2024-03-24T10:08:03.053 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getgid/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.053 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getgid/getgid01.c: Cannot open: Input/output error 2024-03-24T10:08:03.053 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/vhangup: Cannot mkdir: Input/output error 2024-03-24T10:08:03.054 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/vhangup/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.054 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/vhangup/vhangup02.c: Cannot open: Input/output error 2024-03-24T10:08:03.054 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/vhangup/vhangup01.c: Cannot open: Input/output error 2024-03-24T10:08:03.054 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/gethostname: Cannot mkdir: Input/output error 2024-03-24T10:08:03.054 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/gethostname/gethostname01.c: Cannot open: Input/output error 2024-03-24T10:08:03.054 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/gethostname/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.054 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/acct: Cannot mkdir: Input/output error 2024-03-24T10:08:03.054 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/acct/acct02.c: Cannot open: Input/output error 2024-03-24T10:08:03.054 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/acct/acct01.c: Cannot open: Input/output error 2024-03-24T10:08:03.055 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/acct/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.055 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_getscheduler: Cannot mkdir: Input/output error 2024-03-24T10:08:03.055 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_getscheduler/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.055 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_getscheduler/sched_getscheduler02.c: Cannot open: Input/output error 2024-03-24T10:08:03.055 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_getscheduler/sched_getscheduler01.c: Cannot open: Input/output error 2024-03-24T10:08:03.055 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/clone: Cannot mkdir: Input/output error 2024-03-24T10:08:03.055 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/clone/clone02.c: Cannot open: Input/output error 2024-03-24T10:08:03.055 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/clone/clone_platform.h: Cannot open: Input/output error 2024-03-24T10:08:03.055 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/clone/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.056 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/clone/clone04.c: Cannot open: Input/output error 2024-03-24T10:08:03.056 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/clone/clone03.c: Cannot open: Input/output error 2024-03-24T10:08:03.056 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/clone/clone06.c: Cannot open: Input/output error 2024-03-24T10:08:03.056 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/clone/clone07.c: Cannot open: Input/output error 2024-03-24T10:08:03.056 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/clone/clone05.c: Cannot open: Input/output error 2024-03-24T10:08:03.056 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/clone/clone01.c: Cannot open: Input/output error 2024-03-24T10:08:03.056 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setresgid: Cannot mkdir: Input/output error 2024-03-24T10:08:03.056 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setresgid/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.056 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setresgid/setresgid01.c: Cannot open: Input/output error 2024-03-24T10:08:03.056 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setresgid/setresgid02.c: Cannot open: Input/output error 2024-03-24T10:08:03.057 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setresgid/setresgid03.c: Cannot open: Input/output error 2024-03-24T10:08:03.057 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/brk: Cannot mkdir: Input/output error 2024-03-24T10:08:03.057 INFO:tasks.workunit.client.0.smithi022.stderr: 2024-03-24T10:08:03.057 INFO:tasks.workunit.client.0.smithi022.stderr:gzip: tar: ltp-full-20091231/testcases/kernel/syscalls/brk/brk01.c: Cannot open: Input/output error 2024-03-24T10:08:03.057 INFO:tasks.workunit.client.0.smithi022.stderr:stdin: Stale file handle 2024-03-24T10:08:03.057 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/brk/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.057 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setreuid: Cannot mkdir: Input/output error 2024-03-24T10:08:03.057 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setreuid/setreuid03.c: Cannot open: Input/output error 2024-03-24T10:08:03.057 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setreuid/Makefile: Cannot open: Input/output error 2024-03-24T10:08:03.057 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setreuid/setreuid07.c: Cannot open: Input/output error 2024-03-24T10:08:03.058 INFO:tasks.workunit.client.0.smithi022.stderr:tar: Unexpected EOF in archive 2024-03-24T10:08:03.058 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls: Cannot utime: Input/output error 2024-03-24T10:08:03.058 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases/kernel: Cannot utime: Input/output error 2024-03-24T10:08:03.058 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231/testcases: Cannot utime: Input/output error 2024-03-24T10:08:03.058 INFO:tasks.workunit.client.0.smithi022.stderr:tar: ltp-full-20091231: Cannot utime: Input/output error 2024-03-24T10:08:03.058 INFO:tasks.workunit.client.0.smithi022.stderr:tar: Error is not recoverable: exiting now 2024-03-24T10:08:03.058 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-03-24T10:08:03.059 DEBUG:teuthology.orchestra.run.smithi022:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-24T10:08:03.109 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-24T10:08:03.109 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T10:08:04.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:03.823+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:04.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:03.823+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:04.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:03.824+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:04.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:03.824+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:04.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:03.702+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:04.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:03.703+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:04.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:03.703+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:04.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:03.704+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:04.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:03.768+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:04.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:03.768+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:04.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:03.768+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:04.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:03.769+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:05.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:04.680+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:05.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:04.680+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:05.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:04.680+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:05.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:04.681+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:05.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:04.809+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:05.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:04.809+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:05.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:04.809+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:05.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:04.810+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:05.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:04.782+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:05.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:04.782+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:05.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:04.782+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:05.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:04.783+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:05.700 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:08:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:08:05.409+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 17085 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:08:06.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:05.744+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:06.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:05.744+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:06.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:05.744+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:06.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:05.745+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:06.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:05.823+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:06.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:05.823+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:06.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:05.823+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:06.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:05.823+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:06.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:05.700+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:06.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:05.700+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:06.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:05.700+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:06.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:05.701+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:07.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:06.745+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:07.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:06.745+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:07.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:06.745+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:07.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:06.746+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:07.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:06.747+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:07.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:06.747+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:07.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:06.747+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:07.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:06.748+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:07.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:06.814+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:07.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:06.814+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:07.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:06.814+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:07.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:06.815+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:08.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:07.763+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:08.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:07.763+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:08.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:07.763+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:08.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:07.765+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:08.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:07.777+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:08.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:07.777+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:08.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:07.777+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:08.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:07.778+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:08.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:07.806+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:08.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:07.806+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:08.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:07.806+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:08.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:07.807+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:09.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:08.804+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:09.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:08.804+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:09.035 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:08.804+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:09.035 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:08.805+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:09.035 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:08.738+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:09.036 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:08.738+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:09.036 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:08.738+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:09.036 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:08.739+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:09.036 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:08.816+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:09.037 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:08.816+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:09.037 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:08.816+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:09.037 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:08.816+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:10.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:09.788+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:10.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:09.788+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:10.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:09.788+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:10.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:09.789+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:10.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:09.773+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:10.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:09.773+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:10.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:09.773+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:10.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:09.774+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:10.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:09.789+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:10.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:09.789+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:10.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:09.789+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:10.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:09.790+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:10.422 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:08:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:08:10.410+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 17173 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:08:11.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:10.790+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:11.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:10.790+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:11.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:10.790+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:11.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:10.791+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:11.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:10.765+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:11.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:10.765+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:11.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:10.765+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:11.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:10.766+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:11.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:10.837+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:11.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:10.837+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:11.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:10.837+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:11.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:10.837+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:12.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:11.747+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:12.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:11.747+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:12.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:11.747+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:12.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:11.748+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:12.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:11.818+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:12.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:11.818+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:12.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:11.818+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:12.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:11.819+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:12.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:11.766+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:12.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:11.766+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:12.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:11.766+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:12.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:11.767+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:13.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:12.736+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:13.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:12.736+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:13.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:12.736+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:13.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:12.737+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:13.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:12.753+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:13.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:12.753+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:13.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:12.753+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:13.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:12.753+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:13.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:12.834+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:13.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:12.834+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:13.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:12.834+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:13.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:12.835+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:13.858 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:13.732+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:13.859 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:13.732+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:13.859 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:13.732+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:13.859 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:13.733+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:13.860 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:13.742+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:13.860 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:13.742+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:13.860 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:13.742+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:13.860 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:13.743+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:14.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:13.858+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:14.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:13.858+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:14.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:13.858+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:14.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:13.858+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:15.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:14.744+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:15.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:14.745+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:15.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:14.745+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:15.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:14.745+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:15.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:14.782+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:15.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:14.782+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:15.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:14.782+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:15.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:14.783+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:15.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:14.889+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:15.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:14.889+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:15.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:14.889+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:15.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:14.890+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:15.719 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:08:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:08:15.412+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 17264 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:08:16.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:15.915+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:16.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:15.915+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:16.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:15.915+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:16.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:15.916+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:16.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:15.784+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:16.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:15.784+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:16.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:15.784+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:16.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:15.785+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:16.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:15.719+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:16.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:15.719+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:16.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:15.719+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:16.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:15.720+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:17.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:16.763+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:17.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:16.763+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:17.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:16.763+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:17.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:16.763+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:17.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:16.808+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:17.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:16.808+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:17.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:16.808+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:17.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:16.809+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:17.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:16.921+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:17.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:16.921+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:17.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:16.921+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:17.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:16.922+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:18.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:17.958+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:18.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:17.958+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:18.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:17.958+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:18.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:17.959+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:18.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:17.797+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:18.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:17.797+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:18.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:17.797+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:18.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:17.798+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:18.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:17.767+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:18.035 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:17.767+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:18.035 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:17.768+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:18.035 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:17.769+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:19.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:18.794+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:19.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:18.794+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:19.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:18.794+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:19.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:18.794+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:19.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:18.795+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:19.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:18.795+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:19.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:18.795+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:19.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:18.796+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:19.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:18.911+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:19.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:18.911+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:19.285 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:18.911+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:19.285 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:18.911+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:20.162 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:19.874+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:20.163 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:19.874+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:20.163 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:19.874+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:20.163 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:19.875+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:20.164 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:19.846+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:20.164 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:19.846+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:20.164 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:19.846+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:20.164 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:19.847+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:20.165 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:19.830+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:20.165 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:19.830+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:20.165 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:19.830+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:20.165 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:19.831+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:20.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:08:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:08:20.414+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 17352 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:08:21.200 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:20.800+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:21.201 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:20.800+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:21.201 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:20.800+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:21.201 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:20.801+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:21.201 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:20.911+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:21.202 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:20.911+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:21.202 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:20.911+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:21.202 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:20.912+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:21.202 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:20.850+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:21.203 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:20.850+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:21.203 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:20.850+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:21.203 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:20.850+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:22.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:21.829+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:22.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:21.829+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:22.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:21.829+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:22.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:21.830+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:22.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:21.821+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:22.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:21.821+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:22.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:21.821+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:22.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:21.822+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:22.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:21.904+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:22.285 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:21.904+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:22.285 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:21.904+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:22.285 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:21.904+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:23.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:22.869+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:23.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:22.869+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:23.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:22.869+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:23.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:22.870+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:23.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:22.883+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:23.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:22.883+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:23.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:22.883+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:23.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:22.884+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:23.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:22.823+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:23.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:22.823+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:23.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:22.823+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:23.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:22.823+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:24.272 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:23.873+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:24.273 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:23.873+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:24.273 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:23.873+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:24.273 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:23.874+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:24.274 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:23.848+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:24.274 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:23.848+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:24.274 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:23.848+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:24.275 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:23.849+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:24.275 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:23.854+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:24.275 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:23.854+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:24.275 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:23.854+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:24.275 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:23.855+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:25.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:24.907+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:25.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:24.907+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:25.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:24.907+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:25.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:24.908+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:25.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:24.818+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:25.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:24.818+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:25.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:24.818+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:25.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:24.819+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:25.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:24.818+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:25.285 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:24.818+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:25.285 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:24.818+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:25.285 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:24.819+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:25.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:08:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:08:25.416+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 17443 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:08:26.281 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:25.946+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:26.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:25.946+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:26.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:25.946+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:26.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:25.947+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:26.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:25.840+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:26.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:25.840+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:26.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:25.840+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:26.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:25.841+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:26.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:25.868+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:26.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:25.868+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:26.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:25.868+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:26.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:25.869+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:27.071 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:26.810+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:27.071 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:26.810+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:27.071 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:26.810+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:27.071 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:26.811+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:27.071 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:26.884+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:27.072 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:26.884+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:27.072 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:26.884+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:27.072 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:26.885+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:27.072 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:26.949+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:27.072 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:26.949+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:27.072 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:26.949+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:27.072 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:26.949+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:28.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:27.850+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:28.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:27.850+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:28.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:27.850+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:28.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:27.851+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:28.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:27.985+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:28.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:27.985+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:28.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:27.985+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:28.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:27.986+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:28.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:27.850+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:28.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:27.850+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:28.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:27.850+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:28.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:27.851+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:29.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:28.981+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:29.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:28.981+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:29.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:28.981+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:29.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:28.982+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:29.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:28.893+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:29.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:28.893+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:29.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:28.893+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:29.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:28.894+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:29.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:28.812+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:29.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:28.812+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:29.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:28.812+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:29.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:28.813+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:30.163 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:30.005+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:30.163 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:30.005+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:30.164 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:30.005+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:30.164 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:30.005+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:30.164 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:29.830+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:30.164 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:29.830+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:30.164 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:29.830+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:30.164 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:29.831+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:30.165 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:29.862+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:30.165 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:29.863+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:30.165 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:29.863+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:30.165 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:29.863+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:30.781 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:08:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:08:30.418+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 17531 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:08:31.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:30.869+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:31.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:30.869+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:31.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:30.869+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:31.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:30.869+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:31.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:31.019+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:31.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:31.019+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:31.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:31.019+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:31.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:31.020+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:31.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:30.812+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:31.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:30.812+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:31.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:30.812+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:31.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:30.813+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:32.059 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:31.789+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:32.059 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:31.789+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:32.059 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:31.789+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:32.060 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:31.790+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:32.060 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:31.879+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:32.060 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:31.879+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:32.060 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:31.879+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:32.060 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:31.880+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:32.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:32.058+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:32.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:32.058+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:32.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:32.058+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:32.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:32.059+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:33.078 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:32.921+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:33.078 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:32.921+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:33.078 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:32.921+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:33.078 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:32.921+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:33.079 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:32.812+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:33.079 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:32.812+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:33.079 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:32.812+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:33.079 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:32.813+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:33.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:33.077+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:33.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:33.077+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:33.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:33.077+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:33.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:33.078+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:33.892 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:33.780+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:33.892 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:33.780+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:33.892 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:33.780+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:33.893 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:33.781+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:34.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:34.086+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:34.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:34.086+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:34.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:34.086+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:34.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:34.087+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:34.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:33.892+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:34.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:33.892+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:34.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:33.892+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:34.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:33.893+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:35.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:34.940+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:35.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:34.940+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:35.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:34.940+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:35.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:34.941+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:35.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:34.759+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:35.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:34.759+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:35.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:34.759+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:35.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:34.760+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:35.421 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:35.072+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:35.421 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:35.072+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:35.421 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:35.072+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:35.421 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:35.073+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:35.736 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:08:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:08:35.420+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 17621 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:08:36.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:35.736+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:36.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:35.736+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:36.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:35.736+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:36.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:35.737+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:36.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:35.963+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:36.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:35.963+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:36.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:35.963+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:36.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:35.963+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:36.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:36.082+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:36.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:36.082+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:36.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:36.082+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:36.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:36.083+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:36.916 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:36.760+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:36.916 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:36.760+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:36.917 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:36.760+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:36.917 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:36.761+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:37.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:37.114+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:37.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:37.114+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:37.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:37.114+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:37.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:37.115+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:37.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:36.916+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:37.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:36.916+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:37.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:36.916+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:37.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:36.917+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:38.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:37.746+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:38.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:37.746+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:38.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:37.746+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:38.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:37.747+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:38.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:37.966+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:38.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:37.966+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:38.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:37.966+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:38.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:37.967+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:38.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:38.140+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:38.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:38.140+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:38.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:38.140+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:38.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:38.141+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:38.974+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:38.974+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:38.974+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:38.974+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:39.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:38.767+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:39.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:38.767+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:39.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:38.767+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:39.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:38.768+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:39.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:39.181+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:39.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:39.181+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:39.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:39.181+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:39.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:39.182+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:40.024 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:39.746+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:40.024 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:39.746+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:40.025 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:39.746+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:40.025 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:39.747+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:40.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:40.182+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:40.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:40.182+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:40.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:40.182+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:40.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:40.183+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:40.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:40.024+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:40.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:40.024+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:40.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:40.024+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:40.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:40.025+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:40.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:08:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:08:40.422+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 17707 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:08:41.140 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:41.018+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:41.140 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:41.018+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:41.140 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:41.018+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:41.140 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:41.019+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:41.140 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:40.785+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:41.141 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:40.785+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:41.141 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:40.785+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:41.141 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:40.785+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:41.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:41.139+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:41.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:41.139+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:41.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:41.139+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:41.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:41.141+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:42.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:41.772+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:42.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:41.772+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:42.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:41.772+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:42.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:41.773+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:42.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:42.006+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:42.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:42.006+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:42.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:42.006+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:42.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:42.007+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:42.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:42.155+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:42.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:42.155+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:42.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:42.155+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:42.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:42.156+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:42.981 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:42.822+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:42.981 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:42.822+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:42.981 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:42.822+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:42.981 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:42.823+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:43.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:43.205+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:43.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:43.205+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:43.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:43.205+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:43.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:43.206+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:43.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:42.980+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:43.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:42.980+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:43.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:42.980+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:43.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:42.981+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:43.945 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:43.779+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:43.945 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:43.779+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:43.945 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:43.779+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:43.945 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:43.780+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:44.255 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:43.944+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:44.255 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:43.944+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:44.255 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:43.944+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:44.256 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:43.945+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:44.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:44.255+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:44.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:44.255+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:44.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:44.255+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:44.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:44.256+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:45.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:44.926+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:45.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:44.926+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:45.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:44.926+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:45.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:44.927+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:45.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:44.771+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:45.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:44.771+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:45.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:44.771+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:45.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:44.772+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:45.305 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:45.254+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:45.305 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:45.254+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:45.305 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:45.254+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:45.305 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:45.255+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:45.716 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:08:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:08:45.424+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 17798 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:08:46.008 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:45.958+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:46.008 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:45.958+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:46.008 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:45.958+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:46.008 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:45.959+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:46.009 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:45.793+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:46.009 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:45.793+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:46.009 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:45.793+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:46.009 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:45.794+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:46.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:46.256+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:46.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:46.256+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:46.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:46.256+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:46.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:46.257+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:47.230 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:46.797+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:47.230 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:46.797+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:47.230 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:46.797+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:47.230 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:46.798+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:47.230 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:46.946+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:47.231 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:46.946+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:47.231 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:46.946+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:47.231 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:46.947+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:47.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:47.233+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:47.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:47.233+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:47.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:47.233+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:47.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:47.233+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:48.220 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:47.979+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:48.220 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:47.979+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:48.221 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:47.979+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:48.221 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:47.980+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:48.221 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:47.843+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:48.221 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:47.843+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:48.222 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:47.843+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:48.222 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:47.844+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:48.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:48.219+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:48.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:48.219+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:48.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:48.219+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:48.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:48.220+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:49.218 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:49.025+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:49.219 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:49.025+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:49.219 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:49.025+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:49.219 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:49.026+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:49.219 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:48.842+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:49.220 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:48.842+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:49.220 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:48.842+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:49.220 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:48.843+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:49.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:49.217+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:49.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:49.217+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:49.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:49.217+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:49.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:49.218+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:50.165 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:50.073+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:50.165 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:50.073+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:50.165 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:50.073+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:50.165 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:50.074+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:50.166 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:49.847+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:50.166 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:49.847+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:50.166 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:49.847+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:50.166 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:49.848+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:50.427 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:50.191+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:50.427 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:50.191+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:50.427 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:50.191+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:50.427 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:50.192+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:50.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:08:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:08:50.426+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 17885 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:08:51.107 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:50.844+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:51.107 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:50.844+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:51.107 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:50.844+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:51.107 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:50.844+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:51.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:51.240+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:51.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:51.240+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:51.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:51.240+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:51.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:51.240+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:51.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:51.107+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:51.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:51.107+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:51.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:51.107+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:51.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:51.108+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:52.116 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:51.859+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:52.116 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:51.859+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:52.116 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:51.859+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:52.116 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:51.859+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:52.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:52.273+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:52.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:52.273+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:52.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:52.273+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:52.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:52.274+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:52.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:52.116+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:52.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:52.116+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:52.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:52.116+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:52.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:52.118+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:53.225 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:53.081+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:53.226 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:53.081+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:53.226 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:53.081+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:53.226 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:53.082+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:53.226 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:52.864+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:53.227 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:52.864+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:53.227 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:52.864+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:53.227 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:52.864+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:53.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:53.226+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:53.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:53.226+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:53.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:53.226+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:53.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:53.227+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:54.110 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:53.848+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:54.110 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:53.848+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:54.110 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:53.848+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:54.110 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:53.849+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:54.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:54.240+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:54.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:54.240+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:54.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:54.240+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:54.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:54.240+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:54.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:54.110+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:54.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:54.110+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:54.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:54.110+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:54.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:54.111+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:55.274 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:55.062+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:55.274 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:55.062+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:55.275 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:55.062+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:55.275 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:55.063+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:55.275 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:54.882+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:55.275 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:54.882+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:55.276 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:54.882+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:55.276 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:54.883+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:55.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:08:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:08:55.429+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 17974 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:08:55.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:55.274+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:55.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:55.274+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:55.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:55.274+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:55.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:55.275+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:56.253 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:55.854+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:56.253 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:55.854+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:56.254 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:55.854+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:56.254 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:55.855+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:56.254 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:56.094+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:56.254 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:56.094+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:56.255 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:56.094+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:56.255 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:56.095+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:56.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:56.252+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:56.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:56.252+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:56.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:56.252+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:56.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:56.253+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:57.135 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:56.810+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:57.135 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:56.810+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:57.136 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:56.810+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:57.136 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:56.810+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:57.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:57.241+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:57.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:57.241+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:57.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:57.241+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:57.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:57.242+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:57.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:57.136+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:57.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:57.136+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:57.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:57.136+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:57.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:57.137+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:58.169 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:57.843+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:58.169 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:57.843+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:58.170 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:57.843+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:58.170 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:57.844+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:58.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:58.207+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:58.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:58.207+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:58.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:58.207+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:58.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:58.208+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:58.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:58.169+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:58.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:58.170+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:58.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:58.170+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:58.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:58.170+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:59.192 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:58.873+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:59.192 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:58.873+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:59.192 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:58.873+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:08:59.192 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:58.874+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:59.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:59.192+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:59.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:59.192+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:59.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:59.192+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:08:59.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:08:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:08:59.193+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:08:59.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:59.229+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:59.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:59.229+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:59.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:59.229+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:08:59.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:08:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:08:59.229+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:00.162 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:59.895+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:00.162 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:59.895+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:00.162 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:59.895+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:00.162 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:08:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:08:59.896+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:00.430 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:00.229+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:00.431 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:00.229+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:00.431 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:00.229+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:00.431 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:00.229+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:00.431 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:00.162+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:00.432 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:00.162+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:00.432 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:00.162+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:00.432 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:00.163+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:00.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:09:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:09:00.431+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 18062 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:09:01.136 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:00.859+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:01.136 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:00.859+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:01.137 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:00.859+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:01.137 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:00.860+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:01.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:01.269+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:01.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:01.269+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:01.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:01.269+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:01.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:01.271+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:01.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:01.168+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:01.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:01.168+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:01.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:01.168+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:01.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:01.169+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:02.194 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:01.891+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:02.194 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:01.891+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:02.195 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:01.891+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:02.195 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:01.892+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:02.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:02.292+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:02.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:02.292+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:02.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:02.292+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:02.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:02.293+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:02.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:02.195+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:02.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:02.195+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:02.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:02.195+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:02.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:02.196+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:03.188 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:02.896+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:03.189 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:02.896+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:03.189 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:02.896+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:03.189 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:02.897+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:03.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:03.305+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:03.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:03.305+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:03.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:03.305+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:03.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:03.305+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:03.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:03.189+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:03.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:03.189+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:03.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:03.189+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:03.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:03.190+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:04.179 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:03.881+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:04.179 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:03.881+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:04.180 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:03.881+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:04.180 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:03.882+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:04.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:04.310+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:04.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:04.310+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:04.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:04.310+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:04.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:04.311+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:04.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:04.180+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:04.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:04.180+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:04.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:04.180+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:04.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:04.181+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:05.202 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:04.892+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:05.202 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:04.892+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:05.202 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:04.892+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:05.203 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:04.893+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:05.454 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:09:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:09:05.432+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 18152 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:09:05.454 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:05.202+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:05.454 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:05.202+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:05.455 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:05.202+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:05.455 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:05.203+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:05.455 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:05.290+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:05.455 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:05.290+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:05.455 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:05.290+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:05.455 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:05.291+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:06.168 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:05.848+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:06.169 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:05.848+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:06.169 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:05.848+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:06.169 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:05.849+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:06.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:06.282+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:06.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:06.282+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:06.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:06.282+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:06.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:06.282+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:06.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:06.168+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:06.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:06.168+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:06.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:06.168+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:06.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:06.169+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:07.158 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:06.802+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:07.158 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:06.802+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:07.158 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:06.802+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:07.158 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:06.802+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:07.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:07.158+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:07.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:07.158+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:07.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:07.158+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:07.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:07.159+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:07.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:07.236+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:07.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:07.236+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:07.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:07.236+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:07.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:07.237+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:08.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:07.766+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:08.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:07.766+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:08.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:07.766+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:08.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:07.767+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:08.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:08.253+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:08.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:08.253+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:08.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:08.253+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:08.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:08.254+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:08.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:08.188+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:08.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:08.188+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:08.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:08.188+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:08.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:08.189+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:09.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:08.726+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:09.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:08.726+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:09.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:08.726+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:09.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:08.727+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:09.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:09.225+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:09.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:09.225+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:09.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:09.225+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:09.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:09.226+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:09.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:09.235+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:09.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:09.235+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:09.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:09.235+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:09.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:09.235+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:10.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:09.692+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:10.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:09.692+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:10.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:09.692+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:10.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:09.693+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:10.434 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:10.200+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:10.435 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:10.200+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:10.435 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:10.200+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:10.435 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:10.200+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:10.435 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:10.267+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:10.436 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:10.267+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:10.436 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:10.267+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:10.436 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:10.268+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:10.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:09:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:09:10.433+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 18240 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:09:10.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:10.670+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:10.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:10.670+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:10.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:10.670+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:10.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:10.670+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:11.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:11.259+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:11.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:11.259+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:11.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:11.259+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:11.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:11.259+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:11.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:11.218+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:11.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:11.218+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:11.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:11.218+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:11.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:11.219+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:12.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:11.655+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:12.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:11.655+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:12.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:11.655+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:12.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:11.655+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:12.435 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:12.281+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:12.435 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:12.281+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:12.435 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:12.281+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:12.435 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:12.282+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:12.436 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:12.180+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:12.436 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:12.180+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:12.436 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:12.180+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:12.436 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:12.181+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:12.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:12.631+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:12.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:12.631+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:12.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:12.631+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:12.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:12.632+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:13.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:13.226+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:13.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:13.226+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:13.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:13.226+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:13.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:13.227+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:13.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:13.284+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:13.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:13.284+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:13.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:13.284+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:13.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:13.285+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:14.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:13.596+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:14.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:13.596+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:14.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:13.596+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:14.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:13.596+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:14.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:14.278+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:14.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:14.278+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:14.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:14.278+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:14.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:14.280+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:14.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:14.227+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:14.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:14.227+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:14.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:14.227+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:14.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:14.228+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:15.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:14.621+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:15.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:14.621+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:15.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:14.621+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:15.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:14.622+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:15.285 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:15.258+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:15.285 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:15.258+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:15.285 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:15.258+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:15.286 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:15.259+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:15.559 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:09:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:09:15.435+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 18329 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:09:15.559 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:15.285+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:15.559 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:15.285+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:15.559 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:15.285+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:15.559 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:15.286+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:16.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:15.584+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:16.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:15.584+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:16.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:15.584+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:16.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:15.585+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:16.496 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:16.248+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:16.496 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:16.248+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:16.496 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:16.248+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:16.497 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:16.249+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:16.497 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:16.219+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:16.497 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:16.219+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:16.497 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:16.219+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:16.497 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:16.220+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:16.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:16.545+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:16.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:16.545+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:16.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:16.545+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:16.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:16.547+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:17.474 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:17.298+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:17.475 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:17.298+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:17.475 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:17.298+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:17.475 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:17.299+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:17.475 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:17.183+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:17.476 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:17.183+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:17.476 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:17.183+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:17.476 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:17.184+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:17.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:17.589+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:17.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:17.589+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:17.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:17.589+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:17.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:17.590+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:18.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:18.195+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:18.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:18.195+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:18.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:18.195+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:18.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:18.196+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:18.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:18.303+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:18.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:18.303+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:18.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:18.303+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:18.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:18.304+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:19.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:18.543+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:19.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:18.543+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:19.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:18.543+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:19.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:18.543+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:19.511 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:19.298+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:19.511 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:19.298+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:19.511 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:19.298+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:19.512 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:19.299+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:19.512 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:19.173+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:19.512 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:19.173+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:19.512 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:19.173+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:19.513 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:19.174+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:19.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:19.510+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:19.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:19.510+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:19.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:19.510+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:19.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:19.512+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:20.438 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:20.128+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:20.439 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:20.128+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:20.439 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:20.128+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:20.439 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:20.129+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:20.439 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:20.258+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:20.439 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:20.258+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:20.440 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:20.258+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:48.266313+0000 (oldest deadline 2024-03-24T09:59:08.266313+0000) 2024-03-24T10:09:20.440 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:20.259+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:20.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:09:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:09:20.437+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 18417 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:09:20.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:20.520+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:20.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:20.520+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:20.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:20.520+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:20.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:20.521+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:21.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:21.226+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:21.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:21.109+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:21.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:21.109+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:21.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:21.109+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:21.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:21.110+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:22.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:21.560+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:22.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:21.560+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:22.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:21.560+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:22.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:21.560+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:22.514 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:22.236+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:22.515 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:22.099+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:22.515 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:22.099+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:22.515 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:22.099+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:22.515 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:22.100+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:22.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:22.560+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:22.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:22.560+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:22.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:22.560+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:22.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:22.561+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:23.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:23.197+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:23.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:23.070+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:23.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:23.070+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:23.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:23.070+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:23.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:23.071+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:24.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:23.579+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:24.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:23.579+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:24.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:23.579+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:24.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:23.580+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:24.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:24.169+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:24.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:24.047+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:24.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:24.047+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:24.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:24.047+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:24.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:24.048+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:25.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:24.532+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:25.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:24.532+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:25.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:24.532+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:25.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:24.533+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:25.440 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:25.086+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:25.440 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:25.086+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:25.440 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:25.086+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:25.440 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:25.086+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:25.441 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:25.157+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:25.712 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:09:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:09:25.439+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 18507 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:09:25.713 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:25.507+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:25.713 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:25.507+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:25.713 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:25.507+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:25.713 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:25.508+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:26.484 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:26.128+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:26.485 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:26.049+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:26.485 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:26.049+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:26.485 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:26.049+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:26.485 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:26.050+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:26.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:26.484+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:26.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:26.484+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:26.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:26.484+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:26.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:26.484+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:27.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:27.164+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:27.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:27.013+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:27.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:27.013+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:27.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:27.013+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:27.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:27.014+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:27.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:27.500+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:27.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:27.500+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:27.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:27.500+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:27.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:27.501+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:28.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:28.189+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:28.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:28.007+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:28.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:28.007+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:28.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:28.007+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:53.175749+0000 (oldest deadline 2024-03-24T09:59:13.175749+0000) 2024-03-24T10:09:28.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:28.008+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:28.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:28.507+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:28.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:28.507+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:28.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:28.507+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:28.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:28.508+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:29.235 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:28.960+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:29.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:29.235+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:29.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:29.480+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:29.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:29.480+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:29.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:29.480+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:29.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:29.480+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:30.167 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:29.982+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:30.442 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:30.189+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:30.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:09:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:09:30.441+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 18595 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:09:30.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:30.446+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:30.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:30.446+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:30.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:30.446+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T09:58:59.916207+0000 (oldest deadline 2024-03-24T09:59:19.916207+0000) 2024-03-24T10:09:30.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:30.447+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:31.201 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:30.945+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:31.475 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:31.201+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:31.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:31.475+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:32.242 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:31.922+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:32.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:32.242+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:32.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:32.431+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:33.269 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:32.900+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:33.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:33.269+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:33.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:33.477+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:34.244 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:33.909+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:34.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:34.244+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:34.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:34.441+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:35.254 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:34.889+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:35.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:09:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:09:35.443+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 18683 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:09:35.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:35.253+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:35.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:35.487+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:36.271 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:35.897+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:36.526 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:36.270+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:36.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:36.526+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:37.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:36.910+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:37.635 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:37.479+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:37.635 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:37.312+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:38.270 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:37.890+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:38.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:38.270+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:38.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:38.466+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:39.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:38.884+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:39.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:39.305+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:39.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:39.469+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:40.168 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:39.925+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:40.445 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:40.287+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:40.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:09:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:09:40.445+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 18770 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:09:40.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:40.496+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:41.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:40.911+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:41.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:41.544+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:41.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:41.332+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:42.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:41.904+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:42.680 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:42.518+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:42.681 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:42.354+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:42.681 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:42.354+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:42.681 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:42.354+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:42.681 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:42.355+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:43.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:42.936+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:43.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:43.492+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:43.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:43.375+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:43.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:43.375+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:43.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:43.375+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:43.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:43.376+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:44.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:43.960+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:44.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:44.506+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:44.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:44.333+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:44.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:44.333+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:44.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:44.333+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:44.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:44.334+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:45.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:44.973+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:45.572 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:09:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:09:45.447+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 18859 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:09:45.573 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:45.476+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:45.573 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:45.310+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:45.573 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:45.310+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:45.574 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:45.310+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:45.574 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:45.310+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:46.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:45.941+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:46.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:46.451+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:46.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:46.344+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:46.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:46.344+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:46.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:46.344+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:46.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:46.345+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:47.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:46.940+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:47.722 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:47.452+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:47.722 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:47.389+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:47.723 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:47.389+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:47.723 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:47.389+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:47.723 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:47.390+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:48.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:47.941+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:48.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:48.446+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:48.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:48.416+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:48.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:48.416+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:48.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:48.416+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:48.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:48.417+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:49.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:48.908+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:09:49.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:48.908+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:09:49.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:48.908+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:09:49.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:48.908+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:49.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:49.459+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:49.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:49.457+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:49.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:49.457+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:49.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:49.457+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:49.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:49.457+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:50.168 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:49.929+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:09:50.168 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:49.929+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:09:50.169 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:49.929+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:09:50.169 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:49.930+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:50.450 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:50.416+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:50.450 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:50.416+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:50.450 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:50.416+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:50.450 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:50.417+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:50.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:09:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:09:50.449+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 18946 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:09:50.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:50.479+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:51.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:50.945+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:09:51.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:50.945+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:09:51.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:50.945+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:09:51.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:50.947+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:51.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:51.449+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:51.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:51.467+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:51.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:51.467+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:51.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:51.467+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:51.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:51.468+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:52.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:51.934+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:09:52.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:51.934+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:09:52.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:51.934+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:09:52.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:51.935+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:52.761 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:52.417+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:09:52.761 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:52.417+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:09:52.761 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:52.417+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:09:52.761 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:52.418+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:52.762 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:52.434+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:52.762 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:52.434+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:52.762 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:52.434+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:52.762 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:52.435+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:53.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:52.932+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:09:53.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:52.932+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:09:53.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:52.932+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:09:53.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:52.933+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:53.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:53.452+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:53.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:53.452+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:53.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:53.452+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:53.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:53.453+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:53.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:53.449+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:09:53.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:53.449+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:09:53.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:53.449+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:09:53.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:53.450+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:54.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:53.944+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:09:54.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:53.944+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:09:54.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:53.944+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:09:54.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:53.944+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:54.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:54.490+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:54.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:54.490+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:54.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:54.490+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:54.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:54.491+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:54.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:54.440+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:09:54.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:54.440+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:09:54.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:54.440+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:09:54.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:54.441+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:55.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:54.950+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:09:55.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:54.950+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:09:55.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:54.950+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:09:55.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:54.951+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:55.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:09:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:09:55.451+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 19034 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:09:55.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:55.472+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:09:55.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:55.472+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:09:55.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:55.472+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:09:55.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:55.473+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:55.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:55.525+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:55.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:55.525+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:55.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:55.525+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:55.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:55.526+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:56.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:55.932+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:09:56.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:55.932+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:09:56.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:55.933+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:09:56.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:55.933+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:56.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:56.554+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:56.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:56.554+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:56.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:56.554+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:56.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:56.554+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:56.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:56.515+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:09:56.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:56.515+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:09:56.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:56.515+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:09:56.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:56.516+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:57.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:56.972+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:09:57.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:56.972+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:09:57.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:56.972+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:09:57.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:56.974+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:57.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:57.575+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:57.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:57.575+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:57.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:57.575+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:57.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:57.576+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:57.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:57.520+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:09:57.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:57.520+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:09:57.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:57.520+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:09:57.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:57.520+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:58.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:57.931+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:09:58.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:57.931+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:09:58.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:57.931+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:09:58.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:57.932+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:58.896 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:58.595+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:58.896 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:58.595+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:58.896 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:58.595+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:58.896 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:58.596+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:58.897 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:58.545+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:09:58.897 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:58.545+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:09:58.897 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:58.545+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:09:58.897 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:58.546+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:59.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:58.895+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:09:59.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:58.895+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:09:59.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:58.895+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:09:59.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:58.896+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:59.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:59.527+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:09:59.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:59.527+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:09:59.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:59.527+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:09:59.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:09:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:09:59.528+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:09:59.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:59.590+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:59.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:59.590+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:59.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:59.590+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:09:59.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:09:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:09:59.591+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:00.169 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:59.891+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:00.169 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:59.891+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:00.169 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:59.891+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:00.170 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:09:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:09:59.892+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:00.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:10:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:10:00.453+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 19121 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:10:00.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:00.492+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:00.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:00.492+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:00.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:00.492+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:00.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:00.493+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:00.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:00.579+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:00.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:00.579+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:00.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:00.579+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:00.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:00.580+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:01.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:00.940+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:01.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:00.940+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:01.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:00.940+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:01.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:00.941+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:01.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:01.552+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:01.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:01.552+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:01.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:01.552+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:01.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:01.553+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:01.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:01.515+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:01.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:01.515+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:01.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:01.515+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:01.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:01.516+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:02.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:01.974+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:02.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:01.974+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:02.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:01.974+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:02.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:01.975+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:02.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:02.496+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:02.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:02.496+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:02.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:02.496+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:02.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:02.497+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:02.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:02.522+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:02.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:02.522+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:02.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:02.522+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:02.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:02.523+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:03.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:02.928+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:03.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:02.928+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:03.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:02.928+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:03.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:02.929+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:03.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:03.523+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:03.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:03.523+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:03.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:03.523+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:03.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:03.524+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:03.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:03.461+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:03.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:03.461+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:03.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:03.461+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:03.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:03.461+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:04.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:03.934+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:04.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:03.934+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:04.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:03.934+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:04.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:03.935+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:04.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:04.492+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:04.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:04.492+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:04.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:04.492+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:04.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:04.493+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:04.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:04.542+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:04.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:04.542+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:04.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:04.542+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:04.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:04.542+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:05.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:04.937+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:05.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:04.937+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:05.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:04.937+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:05.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:04.938+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:05.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:10:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:10:05.456+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 19209 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:10:05.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:05.535+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:05.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:05.535+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:05.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:05.535+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:05.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:05.536+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:05.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:05.546+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:05.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:05.546+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:05.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:05.546+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:05.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:05.547+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:06.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:05.934+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:06.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:05.934+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:06.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:05.934+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:06.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:05.935+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:06.977 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:06.571+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:06.977 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:06.571+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:06.977 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:06.571+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:06.977 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:06.571+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:06.978 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:06.548+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:06.978 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:06.548+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:06.978 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:06.548+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:06.978 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:06.548+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:07.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:06.976+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:07.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:06.976+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:07.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:06.976+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:07.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:06.977+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:07.886 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:07.576+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:07.886 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:07.576+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:07.886 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:07.576+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:07.886 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:07.577+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:07.887 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:07.555+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:07.887 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:07.555+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:07.887 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:07.555+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:07.887 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:07.555+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:08.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:08.003+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:08.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:08.003+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:08.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:08.003+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:08.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:08.004+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:09.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:08.543+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:09.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:08.543+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:09.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:08.543+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:09.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:08.544+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:09.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:08.586+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:09.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:08.586+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:09.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:08.586+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:09.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:08.587+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:09.506 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:09.044+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:09.506 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:09.044+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:09.506 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:09.044+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:09.507 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:09.044+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:09.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:09.505+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:09.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:09.505+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:09.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:09.505+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:09.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:09.506+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:09.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:09.596+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:09.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:09.596+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:09.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:09.596+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:09.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:09.597+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:10.169 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:10.056+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:10.170 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:10.056+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:10.170 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:10.056+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:10.171 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:10.057+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:10.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:10:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:10:10.458+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 19296 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:10:10.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:10.470+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:10.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:10.470+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:10.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:10.470+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:10.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:10.471+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:10.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:10.586+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:10.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:10.586+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:10.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:10.586+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:10.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:10.587+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:11.441 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:11.095+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:11.441 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:11.095+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:11.441 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:11.095+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:11.442 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:11.096+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:11.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:11.550+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:11.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:11.550+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:11.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:11.550+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:11.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:11.551+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:11.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:11.440+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:11.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:11.440+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:11.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:11.440+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:11.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:11.442+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:12.413 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:12.092+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:12.413 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:12.092+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:12.413 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:12.092+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:12.414 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:12.093+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:12.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:12.412+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:12.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:12.412+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:12.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:12.412+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:12.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:12.413+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:12.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:12.548+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:12.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:12.548+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:12.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:12.548+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:12.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:12.549+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:13.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:13.115+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:13.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:13.115+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:13.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:13.115+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:13.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:13.115+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:13.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:13.410+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:13.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:13.410+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:13.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:13.410+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:13.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:13.411+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:13.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:13.575+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:13.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:13.575+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:13.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:13.575+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:13.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:13.575+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:14.382 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:14.091+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:14.382 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:14.091+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:14.382 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:14.091+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:14.383 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:14.091+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:14.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:14.597+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:14.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:14.597+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:14.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:14.597+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:14.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:14.598+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:14.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:14.382+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:14.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:14.382+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:14.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:14.382+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:14.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:14.382+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:15.419 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:15.141+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:15.420 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:15.141+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:15.420 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:15.141+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:15.420 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:15.142+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:15.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:10:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:10:15.461+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 19384 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:10:15.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:15.420+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:15.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:15.420+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:15.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:15.420+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:15.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:15.421+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:15.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:15.550+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:15.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:15.550+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:15.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:15.550+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:15.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:15.550+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:16.405 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:16.118+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:16.405 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:16.118+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:16.405 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:16.118+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:16.406 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:16.118+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:16.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:16.564+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:16.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:16.564+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:16.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:16.564+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:16.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:16.565+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:16.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:16.406+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:16.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:16.406+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:16.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:16.406+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:16.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:16.407+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:17.387 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:17.084+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:17.387 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:17.084+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:17.387 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:17.084+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:17.387 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:17.085+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:17.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:17.387+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:17.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:17.387+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:17.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:17.387+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:17.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:17.388+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:17.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:17.575+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:17.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:17.575+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:17.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:17.575+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:17.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:17.576+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:18.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:18.058+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:18.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:18.059+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:18.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:18.059+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:18.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:18.059+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:18.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:18.430+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:18.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:18.430+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:18.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:18.430+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:18.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:18.431+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:18.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:18.554+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:18.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:18.554+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:18.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:18.554+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:18.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:18.554+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:19.390 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:19.082+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:19.390 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:19.082+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:19.391 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:19.082+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:19.391 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:19.083+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:19.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:19.390+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:19.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:19.390+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:19.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:19.390+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:19.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:19.391+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:19.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:19.508+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:19.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:19.508+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:19.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:19.508+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:19.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:19.509+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:20.170 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:20.083+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:20.170 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:20.083+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:20.170 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:20.083+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:20.170 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:20.084+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:20.463 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:20.344+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:20.463 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:20.344+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:20.463 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:20.344+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:20.463 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:20.345+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:20.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:10:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:10:20.463+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 19471 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:10:20.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:20.541+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:20.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:20.541+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:20.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:20.541+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:20.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:20.542+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:21.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:21.069+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:21.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:21.069+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:21.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:21.069+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:21.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:21.069+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:21.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:21.361+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:21.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:21.361+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:21.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:21.361+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:21.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:21.361+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:21.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:21.579+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:21.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:21.579+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:21.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:21.579+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:21.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:21.580+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:22.379 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:22.078+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:22.379 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:22.078+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:22.380 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:22.078+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:22.380 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:22.079+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:22.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:22.379+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:22.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:22.379+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:22.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:22.379+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:22.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:22.380+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:22.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:22.612+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:22.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:22.612+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:22.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:22.612+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:22.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:22.613+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:23.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:23.037+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:23.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:23.037+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:23.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:23.037+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:23.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:23.038+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:23.600 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:23.346+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:23.600 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:23.346+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:23.600 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:23.346+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:23.600 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:23.347+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:23.999 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:23.600+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:23.999 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:23.600+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:23.999 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:23.600+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:23.999 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:23.601+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:24.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:23.999+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:24.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:23.999+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:24.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:23.999+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:24.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:24.000+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:24.564 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:24.328+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:24.564 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:24.328+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:24.564 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:24.328+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:24.564 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:24.329+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:25.009 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:24.564+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:25.009 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:24.564+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:25.009 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:24.564+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:25.009 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:24.565+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:25.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:25.009+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:25.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:25.009+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:25.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:25.009+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:25.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:25.010+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:25.588 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:10:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:10:25.465+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 19562 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:10:25.588 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:25.288+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:25.588 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:25.288+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:25.588 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:25.288+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:25.589 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:25.289+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:26.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:25.613+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:26.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:25.613+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:26.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:25.613+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:26.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:25.614+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:26.332 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:26.039+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:26.332 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:26.039+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:26.333 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:26.039+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:26.333 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:26.040+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:26.633 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:26.333+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:26.633 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:26.333+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:26.633 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:26.333+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:26.633 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:26.334+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:27.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:26.634+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:27.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:26.634+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:27.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:26.634+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:27.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:26.635+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:27.369 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:27.040+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:27.369 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:27.040+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:27.369 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:27.040+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:27.369 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:27.041+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:27.652 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:27.369+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:27.652 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:27.369+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:27.652 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:27.369+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:27.653 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:27.370+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:28.018 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:27.652+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:28.018 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:27.652+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:28.018 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:27.652+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:28.018 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:27.652+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:28.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:28.017+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:28.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:28.017+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:28.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:28.017+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:28.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:28.018+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:28.653 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:28.361+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:28.653 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:28.361+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:28.654 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:28.361+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:28.654 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:28.362+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:28.988 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:28.653+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:28.988 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:28.653+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:28.988 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:28.653+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:28.988 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:28.654+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:29.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:28.988+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:29.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:28.988+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:29.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:28.988+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:29.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:28.988+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:29.663 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:29.399+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:29.663 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:29.399+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:29.663 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:29.399+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:29.663 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:29.400+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:30.012 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:29.663+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:30.012 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:29.663+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:30.012 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:29.663+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:30.013 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:29.664+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:30.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:30.012+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:30.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:30.012+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:30.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:30.012+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:30.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:30.012+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:30.689 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:10:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:10:30.466+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 19651 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:10:30.689 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:30.395+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:30.690 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:30.395+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:30.690 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:30.395+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:30.690 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:30.395+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:31.000 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:30.689+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:31.000 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:30.689+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:31.000 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:30.689+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:31.001 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:30.690+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:31.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:30.999+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:31.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:30.999+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:31.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:30.999+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:31.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:31.001+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:31.688 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:31.373+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:31.688 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:31.373+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:31.689 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:31.373+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:31.689 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:31.375+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:32.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:31.688+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:32.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:31.688+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:32.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:31.688+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:32.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:31.688+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:32.398 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:32.049+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:32.416 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:32.049+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:32.416 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:32.049+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:32.416 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:32.050+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:32.707 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:32.397+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:32.707 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:32.397+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:32.707 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:32.397+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:32.707 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:32.398+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:33.002 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:32.706+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:33.002 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:32.706+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:33.002 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:32.706+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:33.002 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:32.708+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:33.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:33.001+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:33.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:33.001+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:33.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:33.001+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:33.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:33.002+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:33.714 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:33.361+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:33.714 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:33.361+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:33.715 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:33.361+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:33.715 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:33.362+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:34.027 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:33.714+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:34.027 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:33.714+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:34.027 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:33.714+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:34.027 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:33.715+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:34.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:34.026+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:34.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:34.026+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:34.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:34.026+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:34.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:34.027+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:34.764 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:34.380+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:34.765 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:34.380+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:34.765 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:34.380+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:34.765 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:34.381+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:35.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:34.764+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:35.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:34.764+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:35.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:34.764+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:35.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:34.765+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:35.348 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:35.071+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:35.349 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:35.071+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:35.349 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:35.071+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:35.349 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:35.072+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:35.759 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:10:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:10:35.469+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 19740 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:10:35.759 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:35.348+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:35.759 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:35.348+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:35.760 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:35.348+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:35.760 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:35.349+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:36.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:35.758+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:36.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:35.758+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:36.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:35.758+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:36.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:35.759+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:36.368 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:36.066+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:36.368 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:36.066+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:36.369 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:36.066+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:36.369 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:36.067+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:36.712 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:36.392+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:36.712 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:36.392+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:36.712 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:36.392+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:36.713 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:36.393+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:37.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:36.712+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:37.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:36.712+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:37.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:36.712+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:37.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:36.713+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:37.354 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:37.057+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:37.354 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:37.057+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:37.354 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:37.057+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:37.354 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:37.058+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:37.745 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:37.353+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:37.745 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:37.353+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:37.745 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:37.353+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:37.745 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:37.354+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:38.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:37.744+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:38.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:37.744+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:38.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:37.744+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:38.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:37.745+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:38.366 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:38.074+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:38.366 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:38.074+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:38.367 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:38.074+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:38.367 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:38.075+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:38.712 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:38.366+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:38.712 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:38.366+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:38.712 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:38.366+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:38.712 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:38.367+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:39.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:38.711+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:39.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:38.711+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:39.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:38.711+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:39.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:38.712+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:39.382 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:39.113+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:39.382 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:39.114+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:39.383 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:39.114+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:39.383 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:39.114+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:39.714 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:39.382+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:39.714 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:39.382+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:39.714 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:39.382+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:39.715 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:39.382+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:40.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:39.714+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:40.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:39.714+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:40.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:39.714+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:40.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:39.714+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:40.473 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:40.116+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:40.473 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:40.116+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:40.473 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:40.116+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:40.473 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:40.117+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:40.474 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:40.340+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:40.474 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:40.340+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:40.474 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:40.340+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:40.474 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:40.341+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:40.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:10:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:10:40.471+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 19828 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:10:40.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:40.670+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:40.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:40.670+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:40.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:40.670+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:40.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:40.671+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:41.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:41.311+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:41.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:41.311+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:41.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:41.311+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:41.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:41.312+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:41.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:41.163+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:41.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:41.163+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:41.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:41.163+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:41.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:41.164+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:42.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:41.663+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:42.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:41.663+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:42.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:41.663+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:42.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:41.664+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:42.512 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:42.195+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:42.512 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:42.195+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:42.512 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:42.195+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:42.513 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:42.196+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:42.513 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:42.284+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:42.513 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:42.284+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:42.513 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:42.284+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:42.513 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:42.285+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:42.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:42.666+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:42.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:42.666+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:42.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:42.666+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:42.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:42.667+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:43.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:43.309+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:43.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:43.309+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:43.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:43.309+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:43.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:43.310+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:43.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:43.185+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:43.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:43.185+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:43.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:43.185+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:43.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:43.185+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:44.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:43.706+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:44.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:43.706+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:44.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:43.706+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:44.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:43.706+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:44.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:44.345+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:44.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:44.345+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:44.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:44.345+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:44.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:44.346+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:44.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:44.231+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:44.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:44.231+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:44.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:44.231+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:44.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:44.232+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:45.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:44.719+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:45.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:44.719+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:45.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:44.719+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:45.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:44.720+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:45.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:45.282+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:45.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:45.282+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:45.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:45.282+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:45.584 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:10:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:10:45.473+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 19917 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:10:45.585 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:45.283+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:45.585 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:45.342+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:45.585 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:45.342+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:45.585 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:45.342+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:45.585 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:45.343+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:46.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:45.693+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:46.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:45.693+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:46.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:45.693+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:46.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:45.694+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:46.676 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:46.288+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:46.676 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:46.288+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:46.676 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:46.288+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:46.677 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:46.289+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:46.677 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:46.312+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:46.677 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:46.312+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:46.677 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:46.312+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:46.677 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:46.313+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:47.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:46.676+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:47.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:46.676+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:47.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:46.676+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:47.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:46.677+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:47.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:47.265+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:47.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:47.265+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:47.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:47.265+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:47.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:47.266+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:47.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:47.326+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:47.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:47.326+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:47.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:47.326+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:47.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:47.327+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:48.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:47.649+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:48.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:47.649+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:48.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:47.649+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:48.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:47.650+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:48.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:48.239+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:48.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:48.239+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:48.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:48.239+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:48.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:48.240+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:48.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:48.283+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:48.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:48.283+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:48.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:48.283+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:48.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:48.283+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:49.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:48.669+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:49.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:48.669+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:49.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:48.669+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:49.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:48.670+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:49.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:49.192+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:49.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:49.192+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:49.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:49.192+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:49.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:49.193+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:49.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:49.260+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:49.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:49.260+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:49.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:49.260+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:49.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:49.261+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:50.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:49.682+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:50.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:49.682+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:50.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:49.682+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:50.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:49.683+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:50.476 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:50.275+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:50.476 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:50.275+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:50.476 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:50.275+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:50.476 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:50.276+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:50.477 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:50.242+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:50.477 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:50.242+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:50.477 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:50.242+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:50.477 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:50.243+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:50.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:10:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:10:50.475+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 20005 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:10:50.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:50.714+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:50.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:50.714+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:50.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:50.714+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:50.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:50.715+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:51.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:51.255+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:51.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:51.255+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:51.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:51.255+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:51.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:51.256+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:51.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:51.242+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:51.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:51.242+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:51.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:51.242+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:51.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:51.242+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:52.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:51.675+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:52.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:51.675+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:52.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:51.675+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:52.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:51.676+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:52.644 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:52.290+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:52.644 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:52.290+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:52.644 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:52.290+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:52.644 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:52.291+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:52.645 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:52.294+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:52.645 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:52.294+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:52.645 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:52.294+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:52.645 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:52.295+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:53.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:52.643+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:53.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:52.643+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:53.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:52.643+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:53.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:52.644+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:53.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:53.337+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:53.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:53.337+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:53.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:53.337+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:53.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:53.338+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:53.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:53.273+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:53.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:53.273+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:53.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:53.273+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:53.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:53.274+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:54.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:53.652+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:54.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:53.652+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:54.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:53.652+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:54.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:53.653+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:54.638 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:54.363+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:54.638 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:54.363+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:54.638 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:54.363+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:54.638 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:54.364+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:54.639 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:54.313+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:54.639 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:54.313+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:54.639 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:54.313+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:54.639 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:54.314+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:55.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:54.637+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:55.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:54.637+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:55.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:54.637+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:55.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:54.638+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:55.599 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:10:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:10:55.477+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 20095 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:10:55.600 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:55.347+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:55.600 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:55.347+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:55.600 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:55.347+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:55.600 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:55.349+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:55.600 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:55.345+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:55.601 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:55.345+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:55.601 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:55.345+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:55.601 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:55.346+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:56.031 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:55.599+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:56.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:55.599+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:56.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:55.599+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:56.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:55.600+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:56.611 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:56.376+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:56.611 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:56.376+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:56.611 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:56.376+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:56.611 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:56.377+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:56.612 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:56.346+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:56.612 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:56.346+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:56.612 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:56.346+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:56.612 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:56.347+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:57.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:56.611+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:57.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:56.611+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:57.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:56.611+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:57.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:56.612+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:57.601 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:57.345+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:57.601 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:57.345+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:57.601 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:57.345+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:57.602 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:57.346+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:57.602 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:57.333+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:57.602 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:57.333+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:57.602 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:57.333+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:57.602 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:57.335+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:58.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:57.601+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:58.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:57.601+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:58.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:57.601+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:58.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:57.601+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:58.578 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:58.319+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:58.579 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:58.319+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:58.579 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:58.319+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:58.579 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:58.320+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:58.579 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:58.319+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:58.579 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:58.319+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:58.579 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:58.319+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:58.579 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:58.320+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:59.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:58.578+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:59.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:58.578+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:59.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:58.578+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:59.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:58.579+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:59.586 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:59.300+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:59.586 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:59.300+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:59.586 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:59.300+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:10:59.587 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:10:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:10:59.301+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:59.587 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:59.335+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:59.587 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:59.335+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:59.587 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:59.335+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:10:59.587 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:10:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:10:59.337+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:10:59.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:59.585+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:59.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:59.585+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:59.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:59.585+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:10:59.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:10:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:10:59.587+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:00.480 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:00.317+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:00.480 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:00.317+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:00.481 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:00.317+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:00.481 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:00.318+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:00.481 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:00.347+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:00.481 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:00.347+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:00.482 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:00.347+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:00.482 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:00.347+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:00.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:11:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:11:00.479+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 20184 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:11:00.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:00.630+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:00.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:00.630+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:00.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:00.630+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:00.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:00.632+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:01.646 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:01.338+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:01.646 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:01.338+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:01.646 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:01.338+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:01.647 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:01.338+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:01.647 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:01.360+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:01.647 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:01.360+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:01.647 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:01.360+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:01.647 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:01.361+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:02.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:01.645+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:02.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:01.645+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:02.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:01.645+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:02.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:01.646+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:02.650 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:02.366+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:02.651 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:02.366+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:02.651 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:02.366+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:02.651 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:02.366+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:02.651 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:02.332+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:02.651 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:02.332+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:02.651 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:02.332+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:02.651 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:02.333+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:03.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:02.650+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:03.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:02.650+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:03.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:02.650+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:03.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:02.651+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:03.597 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:03.379+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:03.598 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:03.379+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:03.598 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:03.379+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:03.598 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:03.380+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:03.598 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:03.302+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:03.598 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:03.302+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:03.598 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:03.302+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:03.598 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:03.302+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:04.016 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:03.668+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:04.016 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:03.668+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:04.016 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:03.668+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:04.016 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:03.669+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:04.672 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:04.380+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:04.672 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:04.380+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:04.672 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:04.380+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:04.673 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:04.381+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:04.673 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:04.296+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:04.673 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:04.296+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:04.673 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:04.296+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:04.673 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:04.297+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:05.031 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:04.672+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:05.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:04.672+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:05.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:04.672+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:05.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:04.672+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:05.504 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:11:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:11:05.481+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 20275 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:11:05.504 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:05.248+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:05.505 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:05.248+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:05.505 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:05.248+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:05.505 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:05.250+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:05.505 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:05.421+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:05.505 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:05.421+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:05.505 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:05.421+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:05.506 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:05.422+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:05.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:05.641+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:05.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:05.641+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:05.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:05.641+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:05.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:05.642+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:06.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:06.409+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:06.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:06.409+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:06.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:06.409+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:06.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:06.410+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:06.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:06.255+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:06.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:06.255+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:06.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:06.255+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:06.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:06.255+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:07.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:06.677+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:07.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:06.677+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:07.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:06.677+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:07.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:06.678+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:07.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:07.372+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:07.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:07.372+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:07.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:07.372+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:07.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:07.373+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:07.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:07.281+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:07.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:07.281+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:07.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:07.281+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:07.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:07.282+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:08.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:07.684+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:08.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:07.684+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:08.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:07.684+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:08.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:07.685+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:08.663 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:08.406+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:08.663 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:08.406+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:08.663 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:08.406+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:08.664 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:08.407+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:08.664 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:08.292+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:08.664 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:08.292+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:08.664 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:08.292+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:08.665 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:08.292+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:09.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:08.662+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:09.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:08.662+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:09.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:08.662+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:09.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:08.663+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:09.660 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:09.393+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:09.660 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:09.393+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:09.661 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:09.393+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:09.661 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:09.393+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:09.661 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:09.342+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:09.661 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:09.342+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:09.662 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:09.342+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:09.662 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:09.343+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:10.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:09.659+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:10.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:09.659+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:10.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:09.659+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:10.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:09.660+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:10.484 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:10.347+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:10.485 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:10.347+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:10.485 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:10.347+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:10.485 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:10.348+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:10.486 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:10.378+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:10.486 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:10.378+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:10.486 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:10.378+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:10.486 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:10.379+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:10.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:11:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:11:10.484+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 20364 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:11:10.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:10.627+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:10.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:10.627+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:10.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:10.627+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:10.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:10.628+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:11.646 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:11.352+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:11.646 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:11.352+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:11.646 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:11.352+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:11.646 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:11.353+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:11.647 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:11.311+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:11.647 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:11.311+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:11.647 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:11.311+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:11.647 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:11.312+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:12.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:11.645+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:12.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:11.645+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:12.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:11.645+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:12.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:11.646+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:12.648 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:12.317+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:12.648 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:12.317+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:12.648 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:12.317+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:12.648 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:12.318+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:12.649 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:12.353+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:12.649 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:12.353+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:12.649 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:12.353+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:12.649 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:12.354+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:13.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:12.647+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:13.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:12.647+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:13.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:12.647+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:13.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:12.648+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:13.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:13.329+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:13.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:13.329+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:13.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:13.329+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:13.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:13.330+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:13.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:13.360+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:13.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:13.360+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:13.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:13.360+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:13.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:13.360+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:14.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:13.616+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:14.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:13.616+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:14.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:13.616+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:14.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:13.618+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:14.591 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:14.342+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:14.591 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:14.342+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:14.592 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:14.342+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:14.592 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:14.343+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:14.592 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:14.340+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:14.592 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:14.340+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:14.593 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:14.340+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:14.593 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:14.341+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:15.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:14.590+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:15.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:14.590+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:15.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:14.590+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:15.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:14.592+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:15.611 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:11:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:11:15.486+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 20454 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:11:15.611 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:15.312+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:15.611 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:15.312+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:15.611 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:15.312+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:15.612 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:15.313+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:15.612 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:15.342+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:15.612 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:15.342+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:15.612 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:15.342+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:15.612 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:15.343+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:16.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:15.610+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:16.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:15.610+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:16.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:15.610+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:16.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:15.611+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:16.647 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:16.346+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:16.648 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:16.346+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:16.648 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:16.346+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:16.648 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:16.346+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:16.648 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:16.304+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:16.649 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:16.304+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:16.649 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:16.304+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:16.649 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:16.306+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:17.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:16.647+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:17.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:16.647+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:17.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:16.647+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:17.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:16.648+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:17.604 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:17.355+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:17.604 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:17.355+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:17.605 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:17.355+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:17.605 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:17.356+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:17.605 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:17.340+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:17.605 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:17.340+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:17.606 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:17.340+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:17.606 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:17.341+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:18.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:17.603+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:18.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:17.603+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:18.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:17.603+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:18.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:17.604+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:18.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:18.352+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:18.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:18.352+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:18.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:18.352+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:18.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:18.352+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:18.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:18.559+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:18.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:18.559+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:18.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:18.559+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:18.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:18.560+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:18.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:18.405+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:18.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:18.405+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:18.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:18.405+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:18.785 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:18.406+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:19.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:19.443+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:19.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:19.443+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:19.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:19.443+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:19.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:19.443+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:19.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:19.320+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:19.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:19.320+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:19.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:19.320+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:19.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:19.321+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:19.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:19.522+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:19.785 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:19.522+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:19.785 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:19.522+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:19.785 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:19.523+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:20.484 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:20.357+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:20.484 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:20.357+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:20.484 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:20.357+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:20.484 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:20.358+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:20.485 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:20.413+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:20.485 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:20.413+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:20.485 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:20.413+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:20.485 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:20.414+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:20.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:11:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:11:20.488+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 20540 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:11:20.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:20.483+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:20.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:20.483+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:20.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:20.483+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:20.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:20.484+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:21.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:21.395+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:21.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:21.395+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:21.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:21.395+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:21.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:21.396+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:21.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:21.404+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:21.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:21.404+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:21.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:21.404+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:21.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:21.404+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:21.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:21.452+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:21.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:21.452+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:21.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:21.452+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:21.785 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:21.454+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:22.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:22.407+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:22.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:22.407+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:22.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:22.407+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:22.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:22.408+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:22.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:22.377+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:22.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:22.377+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:22.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:22.377+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:22.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:22.378+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:22.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:22.371+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:22.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:22.371+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:22.785 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:22.371+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:22.785 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:22.372+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:23.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:23.399+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:23.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:23.399+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:23.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:23.399+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:23.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:23.399+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:23.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:23.359+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:23.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:23.359+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:23.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:23.359+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:23.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:23.360+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:23.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:23.441+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:23.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:23.441+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:23.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:23.441+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:23.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:23.441+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:24.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:24.378+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:24.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:24.378+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:24.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:24.378+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:24.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:24.379+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:24.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:24.323+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:24.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:24.323+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:24.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:24.323+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:24.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:24.324+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:24.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:24.451+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:24.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:24.451+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:24.785 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:24.451+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:24.785 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:24.452+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:25.612 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:11:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:11:25.490+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 20628 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:11:25.612 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:25.475+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:25.612 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:25.475+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:25.613 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:25.475+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:25.613 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:25.475+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:25.613 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:25.387+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:25.613 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:25.387+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:25.613 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:25.387+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:25.614 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:25.388+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:25.614 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:25.291+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:25.614 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:25.291+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:25.615 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:25.291+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:25.615 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:25.291+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:26.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:26.257+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:26.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:26.257+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:26.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:26.257+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:26.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:26.258+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:26.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:26.482+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:26.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:26.482+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:26.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:26.482+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:26.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:26.483+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:26.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:26.403+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:26.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:26.403+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:26.535 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:26.403+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:26.535 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:26.403+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:27.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:27.494+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:27.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:27.494+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:27.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:27.494+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:27.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:27.495+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:27.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:27.256+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:27.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:27.256+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:27.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:27.256+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:27.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:27.257+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:27.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:27.432+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:27.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:27.432+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:27.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:27.432+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:27.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:27.432+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:28.511 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:28.391+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:28.511 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:28.391+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:28.511 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:28.391+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:28.511 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:28.391+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:28.512 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:28.259+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:28.512 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:28.259+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:28.512 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:28.259+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:28.512 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:28.260+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:28.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:28.511+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:28.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:28.511+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:28.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:28.511+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:28.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:28.511+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:29.541 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:29.363+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:29.541 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:29.363+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:29.541 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:29.363+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:29.541 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:29.363+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:29.541 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:29.290+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:29.542 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:29.290+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:29.542 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:29.290+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:29.542 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:29.291+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:29.853 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:29.540+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:29.853 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:29.540+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:29.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:29.540+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:29.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:29.542+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:30.495 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:30.244+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:30.495 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:30.244+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:30.495 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:30.244+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:30.495 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:30.245+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:30.495 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:30.386+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:30.496 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:30.386+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:30.496 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:30.386+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:30.496 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:30.387+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:30.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:11:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:11:30.492+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 20710 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:11:30.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:30.591+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:30.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:30.591+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:30.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:30.591+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:30.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:30.592+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:31.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:31.237+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:31.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:31.237+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:31.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:31.237+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:31.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:31.237+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:31.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:31.376+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:31.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:31.376+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:31.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:31.376+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:31.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:31.377+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:32.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:31.548+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:32.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:31.548+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:32.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:31.548+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:32.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:31.548+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:32.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:32.230+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:32.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:32.230+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:32.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:32.230+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:32.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:32.231+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:32.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:32.364+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:32.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:32.364+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:32.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:32.364+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:32.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:32.365+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:33.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:32.542+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:33.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:32.542+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:33.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:32.542+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:33.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:32.543+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:33.530 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:33.320+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:33.530 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:33.320+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:33.530 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:33.320+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:33.531 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:33.321+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:33.531 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:33.215+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:33.531 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:33.215+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:33.531 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:33.215+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:33.531 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:33.216+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:33.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:33.529+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:33.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:33.529+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:33.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:33.529+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:33.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:33.531+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:34.509 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:34.365+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:34.510 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:34.365+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:34.510 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:34.365+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:34.510 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:34.365+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:34.510 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:34.199+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:34.511 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:34.199+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:34.511 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:34.199+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:34.511 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:34.199+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:34.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:34.509+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:34.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:34.509+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:34.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:34.509+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:34.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:34.509+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:35.495 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:35.394+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:35.496 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:35.394+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:35.496 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:35.394+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:35.496 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:35.394+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:35.496 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:35.183+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:35.497 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:35.183+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:35.497 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:35.183+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:35.497 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:35.183+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:35.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:11:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:11:35.495+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 20791 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:11:35.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:35.528+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:35.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:35.528+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:35.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:35.528+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:35.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:35.529+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:36.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:36.431+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:36.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:36.431+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:36.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:36.431+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:36.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:36.431+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:36.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:36.231+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:36.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:36.231+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:36.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:36.231+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:36.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:36.232+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:37.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:36.540+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:37.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:36.540+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:37.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:36.540+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:37.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:36.541+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:37.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:37.406+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:37.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:37.406+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:37.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:37.406+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:37.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:37.406+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:37.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:37.241+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:37.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:37.242+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:37.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:37.242+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:37.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:37.242+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:38.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:37.539+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:38.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:37.539+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:38.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:37.539+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:38.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:37.540+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:38.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:38.383+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:38.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:38.383+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:38.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:38.383+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:38.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:38.383+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:38.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:38.281+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:38.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:38.281+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:38.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:38.281+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:38.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:38.282+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:39.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:38.589+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:39.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:38.589+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:39.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:38.589+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:39.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:38.590+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:39.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:39.418+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:39.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:39.418+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:39.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:39.418+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:39.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:39.418+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:39.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:39.239+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:39.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:39.239+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:39.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:39.239+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:39.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:39.239+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:40.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:39.635+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:40.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:39.635+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:40.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:39.635+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:40.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:39.636+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:40.497 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:40.252+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:40.497 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:40.252+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:40.497 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:40.252+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:40.498 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:40.253+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:40.498 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:40.379+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:40.498 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:40.379+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:40.498 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:40.379+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:40.498 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:40.380+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:40.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:11:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:11:40.498+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 20871 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:11:40.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:40.649+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:40.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:40.649+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:40.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:40.649+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:40.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:40.650+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:41.603 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:41.296+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:41.603 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:41.296+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:41.603 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:41.296+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:41.604 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:41.297+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:41.604 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:41.381+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:41.604 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:41.381+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:41.604 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:41.381+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:41.604 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:41.381+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:42.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:41.603+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:42.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:41.603+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:42.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:41.604+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:42.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:41.604+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:42.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:42.268+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:42.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:42.268+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:42.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:42.268+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:42.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:42.269+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:42.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:42.394+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:42.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:42.394+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:42.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:42.394+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:42.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:42.395+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:43.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:42.600+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:43.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:42.600+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:43.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:42.600+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:43.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:42.600+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:43.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:43.240+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:43.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:43.240+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:43.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:43.240+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:43.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:43.241+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:43.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:43.390+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:43.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:43.390+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:43.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:43.390+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:43.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:43.391+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:43.953 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:43.620+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:43.953 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:43.621+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:43.953 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:43.621+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:43.953 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:43.621+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:44.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:44.418+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:44.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:44.418+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:44.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:44.418+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:44.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:44.419+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:44.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:44.264+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:44.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:44.264+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:44.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:44.264+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:44.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:44.265+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:45.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:44.607+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:45.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:44.607+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:45.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:44.607+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:45.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:44.608+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:45.592 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:11:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:11:45.500+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 20956 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:11:45.592 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:45.294+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:45.593 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:45.294+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:45.593 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:45.294+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:45.593 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:45.295+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:45.593 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:45.441+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:45.593 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:45.441+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:45.594 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:45.441+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:45.594 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:45.442+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:46.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:45.592+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:46.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:45.592+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:46.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:45.592+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:46.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:45.593+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:46.639 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:46.487+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:46.639 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:46.487+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:46.639 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:46.487+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:46.639 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:46.487+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:46.640 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:46.329+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:46.640 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:46.329+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:46.640 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:46.329+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:46.640 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:46.330+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:47.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:46.639+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:47.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:46.639+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:47.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:46.639+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:47.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:46.640+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:47.678 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:47.461+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:47.678 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:47.461+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:47.678 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:47.461+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:47.678 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:47.461+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:47.678 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:47.361+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:47.679 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:47.361+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:47.679 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:47.361+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:47.679 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:47.361+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:48.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:47.678+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:48.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:47.678+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:48.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:47.678+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:48.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:47.679+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:48.692 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:48.444+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:48.693 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:48.444+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:48.693 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:48.444+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:48.693 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:48.445+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:48.693 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:48.398+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:48.693 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:48.398+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:48.694 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:48.398+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:48.694 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:48.399+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:49.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:48.693+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:49.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:48.693+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:49.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:48.693+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:49.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:48.694+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:49.654 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:49.396+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:49.654 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:49.396+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:49.655 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:49.396+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:49.655 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:49.397+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:49.655 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:49.406+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:49.655 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:49.406+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:49.655 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:49.406+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:49.656 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:49.407+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:50.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:49.654+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:50.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:49.654+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:50.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:49.654+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:50.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:49.655+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:50.425 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:50.352+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:50.426 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:50.352+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:50.426 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:50.352+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:50.426 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:50.353+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:50.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:11:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:11:50.501+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 21042 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:11:50.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:50.618+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:50.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:50.618+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:50.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:50.618+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:50.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:50.619+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:50.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:50.425+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:50.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:50.425+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:50.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:50.425+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:50.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:50.426+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:51.597 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:51.323+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:51.598 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:51.323+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:51.598 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:51.323+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:51.598 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:51.323+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:51.598 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:51.409+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:51.598 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:51.409+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:51.598 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:51.409+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:51.599 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:51.410+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:52.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:51.597+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:52.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:51.597+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:52.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:51.597+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:52.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:51.598+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:52.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:52.315+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:52.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:52.315+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:52.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:52.315+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:52.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:52.316+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:52.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:52.370+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:52.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:52.370+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:52.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:52.370+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:52.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:52.371+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:52.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:52.549+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:52.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:52.549+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:52.785 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:52.549+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:52.785 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:52.550+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:53.517 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:53.315+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:53.518 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:53.315+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:53.518 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:53.315+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:53.518 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:53.316+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:53.518 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:53.365+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:53.519 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:53.365+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:53.519 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:53.365+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:53.519 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:53.366+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:53.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:53.517+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:53.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:53.517+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:53.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:53.517+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:53.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:53.517+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:54.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:54.413+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:54.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:54.413+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:54.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:54.413+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:54.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:54.414+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:54.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:54.522+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:54.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:54.522+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:54.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:54.522+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:54.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:54.523+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:54.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:54.363+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:54.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:54.363+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:54.785 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:54.363+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:54.785 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:54.364+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:55.626 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:55.372+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:55.626 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:55.372+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:55.626 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:55.372+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:55.626 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:55.373+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:55.627 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:55.450+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:55.627 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:55.450+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:55.627 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:55.450+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:55.627 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:55.450+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:55.627 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:55.540+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:55.627 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:55.540+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:55.628 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:55.540+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:55.628 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:55.541+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:55.628 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:11:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:11:55.504+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 21132 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:11:56.529 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:56.407+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:56.529 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:56.407+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:56.529 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:56.407+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:56.530 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:56.408+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:56.530 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:56.369+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:56.530 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:56.369+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:56.530 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:56.369+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:56.531 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:56.370+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:56.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:56.528+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:56.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:56.528+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:56.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:56.528+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:56.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:56.529+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:57.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:57.418+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:57.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:57.418+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:57.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:57.418+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:57.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:57.418+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:57.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:57.422+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:57.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:57.422+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:57.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:57.422+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:57.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:57.423+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:57.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:57.548+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:57.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:57.548+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:57.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:57.548+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:57.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:57.549+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:58.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:58.458+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:58.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:58.458+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:58.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:58.458+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:58.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:58.459+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:58.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:58.501+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:58.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:58.501+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:58.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:58.501+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:58.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:58.502+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:58.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:58.396+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:58.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:58.396+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:58.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:58.396+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:58.785 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:58.396+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:59.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:59.490+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:59.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:59.490+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:59.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:59.490+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:11:59.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:11:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:11:59.490+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:59.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:59.381+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:59.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:59.381+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:59.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:59.381+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:11:59.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:11:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:11:59.382+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:11:59.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:59.526+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:59.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:59.526+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:59.785 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:59.526+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:11:59.785 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:11:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:11:59.527+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:00.463 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:00.398+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:00.463 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:00.398+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:00.463 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:00.398+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:00.463 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:00.399+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:00.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:12:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:12:00.506+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 21220 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:12:00.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:00.462+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:00.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:00.462+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:00.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:00.462+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:00.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:00.463+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:00.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:00.512+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:00.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:00.512+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:00.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:00.512+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:00.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:00.513+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:01.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:01.560+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:01.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:01.560+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:01.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:01.560+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:01.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:01.561+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:01.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:01.401+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:01.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:01.401+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:01.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:01.401+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:01.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:01.402+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:01.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:01.430+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:01.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:01.430+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:01.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:01.430+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:01.785 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:01.431+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:02.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:02.545+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:02.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:02.545+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:02.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:02.545+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:02.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:02.546+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:02.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:02.407+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:02.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:02.407+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:02.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:02.407+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:02.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:02.408+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:02.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:02.444+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:02.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:02.444+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:02.785 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:02.444+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:02.785 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:02.445+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:03.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:03.590+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:03.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:03.590+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:03.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:03.590+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:03.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:03.591+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:03.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:03.449+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:03.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:03.450+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:03.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:03.450+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:03.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:03.450+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:03.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:03.461+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:03.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:03.461+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:03.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:03.461+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:03.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:03.462+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:04.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:04.558+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:04.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:04.558+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:04.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:04.558+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:04.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:04.559+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:04.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:04.466+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:04.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:04.466+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:04.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:04.466+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:04.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:04.467+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:04.785 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:04.497+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:04.785 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:04.497+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:04.785 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:04.497+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:04.785 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:04.498+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:05.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:12:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:12:05.509+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 21312 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:12:05.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:05.587+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:05.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:05.587+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:05.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:05.587+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:05.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:05.588+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:05.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:05.509+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:05.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:05.509+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:05.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:05.509+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:05.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:05.510+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:05.785 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:05.419+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:05.785 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:05.419+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:05.785 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:05.419+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:05.785 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:05.420+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:06.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:06.561+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:06.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:06.561+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:06.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:06.561+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:06.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:06.563+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:06.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:06.375+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:06.785 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:06.375+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:06.785 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:06.375+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:06.785 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:06.376+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:06.785 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:06.525+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:06.785 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:06.525+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:06.786 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:06.525+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:06.786 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:06.525+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:07.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:07.522+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:07.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:07.522+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:07.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:07.522+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:07.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:07.523+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:07.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:07.611+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:07.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:07.611+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:07.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:07.611+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:07.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:07.612+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:07.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:07.371+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:07.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:07.371+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:07.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:07.371+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:07.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:07.372+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:08.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:08.346+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:08.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:08.346+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:08.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:08.346+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:08.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:08.346+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:08.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:08.494+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:08.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:08.494+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:08.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:08.494+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:08.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:08.495+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:08.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:08.574+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:08.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:08.574+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:08.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:08.574+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:08.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:08.575+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:09.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:09.370+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:09.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:09.370+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:09.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:09.370+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:09.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:09.371+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:09.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:09.545+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:09.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:09.545+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:09.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:09.545+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:09.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:09.546+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:09.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:09.559+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:09.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:09.559+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:09.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:09.559+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:09.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:09.560+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:10.512 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:10.322+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:10.512 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:10.322+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:10.513 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:10.322+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:10.513 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:10.322+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:10.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:12:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:12:10.511+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 21400 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:12:10.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:10.513+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:10.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:10.513+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:10.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:10.513+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:10.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:10.515+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:10.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:10.539+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:10.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:10.539+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:10.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:10.539+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:10.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:10.540+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:11.601 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:11.469+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:11.601 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:11.469+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:11.601 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:11.469+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:11.602 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:11.470+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:11.602 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:11.331+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:11.602 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:11.331+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:11.603 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:11.331+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:11.603 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:11.332+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:11.603 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:11.531+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:11.604 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:11.531+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:11.604 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:11.531+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:11.604 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:11.532+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:12.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:12.374+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:12.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:12.374+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:12.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:12.374+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:12.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:12.375+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:12.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:12.456+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:12.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:12.456+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:12.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:12.456+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:12.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:12.457+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:12.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:12.582+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:12.785 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:12.582+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:12.785 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:12.582+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:12.785 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:12.583+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:13.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:13.621+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:13.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:13.621+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:13.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:13.621+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:13.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:13.622+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:13.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:13.420+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:13.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:13.420+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:13.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:13.420+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:13.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:13.421+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:13.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:13.486+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:13.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:13.486+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:13.785 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:13.486+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:13.785 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:13.486+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:14.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:14.437+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:14.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:14.437+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:14.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:14.437+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:14.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:14.438+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:14.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:14.519+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:14.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:14.519+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:14.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:14.519+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:14.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:14.520+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:14.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:14.617+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:14.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:14.617+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:14.785 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:14.617+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:14.785 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:14.617+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:15.683 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:12:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:12:15.513+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 21491 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:12:15.684 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:15.519+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:15.684 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:15.519+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:15.684 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:15.519+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:15.684 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:15.520+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:15.685 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:15.591+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:15.685 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:15.591+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:15.685 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:15.591+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:15.685 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:15.592+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:15.686 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:15.400+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:15.686 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:15.400+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:15.686 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:15.400+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:15.686 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:15.401+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:16.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:16.600+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:16.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:16.600+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:16.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:16.600+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:16.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:16.601+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:16.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:16.538+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:16.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:16.538+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:16.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:16.538+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:16.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:16.539+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:16.785 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:16.362+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:16.785 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:16.362+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:16.785 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:16.362+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:16.785 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:16.363+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:17.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:17.576+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:17.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:17.576+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:17.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:17.576+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:17.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:17.577+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:17.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:17.344+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:17.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:17.344+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:17.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:17.344+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:17.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:17.345+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:17.785 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:17.519+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:17.785 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:17.519+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:17.785 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:17.519+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:17.785 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:17.519+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:18.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:18.327+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:18.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:18.327+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:18.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:18.327+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:18.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:18.327+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:18.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:18.509+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:18.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:18.509+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:18.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:18.509+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:18.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:18.510+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:18.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:18.539+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:18.785 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:18.539+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:18.785 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:18.539+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:18.785 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:18.540+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:19.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:19.481+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:19.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:19.481+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:19.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:19.481+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:19.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:19.482+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:19.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:19.575+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:19.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:19.575+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:19.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:19.575+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:19.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:19.576+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:19.785 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:19.358+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:19.785 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:19.358+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:19.785 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:19.358+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:19.785 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:19.359+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:20.466 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:20.357+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:20.466 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:20.357+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:20.466 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:20.357+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:20.466 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:20.357+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:20.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:12:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:12:20.516+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 21579 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:12:20.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:20.533+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:20.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:20.533+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:20.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:20.533+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:20.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:20.534+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:20.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:20.465+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:20.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:20.465+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:20.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:20.465+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:20.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:20.466+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:21.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:21.583+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:21.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:21.583+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:21.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:21.583+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:21.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:21.584+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:21.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:21.396+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:21.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:21.396+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:21.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:21.396+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:21.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:21.397+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:21.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:21.498+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:21.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:21.498+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:21.785 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:21.498+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:21.785 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:21.499+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:22.635 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:22.368+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:22.635 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:22.368+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:22.636 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:22.368+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:22.636 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:22.369+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:22.636 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:22.545+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:22.636 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:22.545+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:22.636 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:22.545+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:22.636 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:22.546+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:23.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:22.635+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:23.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:22.635+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:23.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:22.635+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:23.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:22.636+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:23.614 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:23.359+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:23.614 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:23.359+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:23.615 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:23.359+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:23.615 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:23.360+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:23.615 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:23.555+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:23.615 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:23.555+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:23.615 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:23.555+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:23.615 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:23.556+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:23.888 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:23.614+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:23.888 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:23.614+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:23.888 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:23.614+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:23.888 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:23.614+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:24.607 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:24.329+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:24.607 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:24.329+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:24.607 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:24.329+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:24.607 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:24.329+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:24.607 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:24.527+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:24.608 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:24.527+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:24.608 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:24.527+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:24.608 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:24.528+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:25.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:24.606+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:25.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:24.606+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:25.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:24.606+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:25.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:24.607+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:25.641 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:12:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:12:25.518+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 21671 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:12:25.642 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:25.361+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:25.642 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:25.361+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:25.642 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:25.361+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:25.642 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:25.363+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:25.643 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:25.501+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:25.643 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:25.502+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:25.643 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:25.502+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:25.643 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:25.502+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:26.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:25.652+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:26.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:25.652+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:26.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:25.652+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:26.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:25.653+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:26.685 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:26.507+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:26.685 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:26.507+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:26.685 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:26.507+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:26.685 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:26.509+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:26.686 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:26.362+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:26.686 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:26.362+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:26.686 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:26.362+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:26.686 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:26.363+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:27.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:26.684+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:27.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:26.684+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:27.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:26.684+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:27.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:26.685+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:27.707 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:27.328+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:27.707 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:27.328+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:27.707 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:27.328+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:27.708 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:27.329+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:27.708 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:27.496+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:27.708 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:27.496+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:27.708 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:27.496+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:27.708 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:27.496+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:28.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:27.707+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:28.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:27.707+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:28.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:27.707+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:28.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:27.708+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:28.749 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:28.299+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:28.749 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:28.300+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:28.749 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:28.300+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:28.750 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:28.301+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:28.750 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:28.461+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:28.750 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:28.461+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:28.750 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:28.461+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:28.750 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:28.462+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:29.030 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:28.749+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:29.031 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:28.749+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:29.031 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:28.749+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:29.031 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:28.749+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:29.733 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:29.344+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:29.733 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:29.344+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:29.733 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:29.344+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:29.733 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:29.345+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:29.734 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:29.451+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:29.734 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:29.451+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:29.734 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:29.451+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:29.734 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:29.451+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:30.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:29.732+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:30.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:29.732+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:30.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:29.732+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:30.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:29.733+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:30.521 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:30.412+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:30.521 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:30.412+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:30.521 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:30.412+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:30.521 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:30.412+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:30.522 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:30.296+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:30.522 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:30.296+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:30.522 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:30.296+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:30.522 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:30.297+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:30.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:12:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:12:30.520+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 21759 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:12:30.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:30.695+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:30.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:30.695+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:30.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:30.695+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:30.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:30.695+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:31.672 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:31.375+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:31.672 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:31.375+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:31.672 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:31.375+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:31.672 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:31.376+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:31.673 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:31.290+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:31.673 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:31.290+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:31.673 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:31.290+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:31.673 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:31.291+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:32.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:31.671+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:32.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:31.671+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:32.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:31.671+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:32.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:31.672+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:32.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:32.251+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:32.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:32.251+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:32.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:32.251+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:32.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:32.252+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:32.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:32.418+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:32.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:32.418+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:32.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:32.418+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:32.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:32.418+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:33.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:32.702+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:33.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:32.702+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:33.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:32.702+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:33.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:32.703+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:33.460 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:33.238+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:33.461 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:33.238+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:33.461 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:33.238+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:33.461 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:33.239+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:33.714 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:33.460+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:33.714 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:33.460+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:33.714 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:33.460+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:33.714 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:33.461+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:34.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:33.713+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:34.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:33.713+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:34.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:33.713+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:34.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:33.714+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:34.430 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:34.233+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:34.430 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:34.233+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:34.431 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:34.233+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:34.431 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:34.233+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:34.742 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:34.429+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:34.742 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:34.429+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:34.743 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:34.429+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:34.743 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:34.430+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:35.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:34.741+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:35.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:34.741+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:35.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:34.741+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:35.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:34.742+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:35.523 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:35.460+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:35.524 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:35.460+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:35.524 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:35.460+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:35.524 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:35.461+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:35.524 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:35.227+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:35.525 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:35.227+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:35.525 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:35.227+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:35.525 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:35.228+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:35.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:12:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:12:35.523+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 21850 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:12:35.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:35.697+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:35.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:35.697+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:35.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:35.697+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:35.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:35.698+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:36.430 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:36.189+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:36.431 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:36.189+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:36.431 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:36.189+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:36.431 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:36.190+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:36.728 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:36.430+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:36.728 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:36.430+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:36.728 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:36.430+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:36.728 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:36.430+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:37.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:36.727+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:37.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:36.727+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:37.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:36.727+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:37.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:36.728+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:37.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:37.464+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:37.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:37.464+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:37.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:37.464+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:37.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:37.465+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:37.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:37.221+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:37.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:37.221+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:37.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:37.221+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:37.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:37.222+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:38.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:37.680+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:38.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:37.680+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:38.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:37.680+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:38.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:37.681+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:38.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:38.453+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:38.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:38.453+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:38.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:38.453+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:38.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:38.455+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:38.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:38.230+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:38.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:38.230+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:38.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:38.230+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:38.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:38.231+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:39.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:38.656+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:39.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:38.656+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:39.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:38.656+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:39.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:38.656+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:39.496 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:39.274+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:39.496 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:39.274+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:39.496 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:39.274+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:39.497 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:39.274+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:39.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:39.661+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:39.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:39.661+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:39.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:39.661+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:39.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:39.661+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:39.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:39.495+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:39.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:39.495+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:39.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:39.495+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:39.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:39.496+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:40.497 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:40.241+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:40.497 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:40.241+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:40.497 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:40.241+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:40.497 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:40.242+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:40.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:12:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:12:40.525+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 21938 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:12:40.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:40.496+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:40.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:40.496+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:40.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:40.496+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:40.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:40.497+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:40.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:40.700+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:40.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:40.700+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:40.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:40.700+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:40.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:40.701+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:41.726 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:41.508+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:41.726 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:41.508+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:41.727 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:41.508+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:41.727 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:41.509+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:41.727 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:41.284+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:41.727 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:41.284+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:41.727 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:41.284+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:41.727 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:41.285+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:42.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:41.725+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:42.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:41.725+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:42.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:41.725+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:42.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:41.726+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:42.710 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:42.543+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:42.710 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:42.543+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:42.710 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:42.543+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:42.710 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:42.544+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:42.711 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:42.334+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:42.711 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:42.334+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:42.711 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:42.334+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:42.711 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:42.335+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:43.011 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:42.709+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:43.011 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:42.709+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:43.012 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:42.709+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:43.012 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:42.710+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:43.553 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:43.301+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:43.553 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:43.301+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:43.554 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:43.301+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:43.554 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:43.302+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:44.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:43.552+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:44.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:43.552+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:44.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:43.552+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:44.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:43.553+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:44.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:43.663+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:44.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:43.663+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:44.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:43.663+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:44.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:43.664+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:44.530 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:44.278+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:44.531 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:44.278+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:44.531 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:44.278+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:44.531 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:44.279+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:44.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:44.529+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:44.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:44.529+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:44.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:44.529+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:44.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:44.529+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:44.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:44.709+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:44.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:44.709+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:44.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:44.709+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:44.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:44.709+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:45.528 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:45.234+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:45.528 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:45.234+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:45.528 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:45.234+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:45.528 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:45.235+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:45.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:12:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:12:45.527+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 22030 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:12:45.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:45.562+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:45.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:45.562+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:45.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:45.562+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:45.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:45.563+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:45.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:45.749+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:45.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:45.749+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:45.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:45.749+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:45.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:45.750+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:46.523 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:46.192+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:46.523 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:46.192+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:46.523 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:46.192+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:46.523 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:46.193+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:46.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:46.522+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:46.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:46.522+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:46.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:46.522+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:46.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:46.523+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:46.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:46.719+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:46.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:46.719+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:46.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:46.719+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:46.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:46.720+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:47.491 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:47.191+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:47.491 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:47.191+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:47.491 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:47.191+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:47.491 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:47.192+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:47.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:47.490+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:47.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:47.490+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:47.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:47.490+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:47.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:47.491+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:47.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:47.720+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:47.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:47.720+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:47.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:47.720+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:47.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:47.721+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:48.505 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:48.224+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:48.505 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:48.224+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:48.505 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:48.224+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:48.505 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:48.225+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:48.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:48.723+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:48.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:48.723+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:48.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:48.723+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:48.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:48.723+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:48.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:48.504+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:48.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:48.504+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:48.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:48.504+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:48.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:48.505+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:49.492 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:49.195+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:49.493 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:49.195+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:49.493 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:49.195+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:49.493 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:49.196+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:49.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:49.492+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:49.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:49.492+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:49.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:49.492+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:49.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:49.493+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:49.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:49.726+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:49.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:49.726+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:49.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:49.726+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:49.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:49.727+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:50.510 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:50.207+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:50.511 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:50.207+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:50.511 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:50.207+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:50.511 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:50.208+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:50.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:12:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:12:50.530+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 22118 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:12:50.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:50.741+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:50.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:50.741+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:50.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:50.741+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:50.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:50.743+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:50.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:50.510+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:50.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:50.510+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:50.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:50.510+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:50.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:50.511+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:51.467 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:51.182+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:51.467 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:51.182+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:51.467 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:51.182+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:51.467 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:51.182+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:51.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:51.466+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:51.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:51.466+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:51.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:51.466+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:51.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:51.467+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:52.164 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:51.791+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:52.165 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:51.791+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:52.165 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:51.791+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:52.165 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:51.792+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:52.443 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:52.164+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:52.443 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:52.164+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:52.443 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:52.164+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:52.443 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:52.165+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:52.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:52.442+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:52.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:52.442+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:52.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:52.442+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:52.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:52.443+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:53.172 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:52.836+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:53.172 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:52.836+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:53.172 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:52.836+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:53.172 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:52.837+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:53.440 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:53.171+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:53.441 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:53.171+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:53.441 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:53.171+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:53.441 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:53.172+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:53.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:53.440+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:53.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:53.440+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:53.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:53.440+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:53.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:53.441+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:54.217 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:53.853+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:54.217 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:53.853+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:54.217 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:53.853+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:54.217 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:53.853+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:54.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:54.217+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:54.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:54.217+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:54.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:54.217+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:54.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:54.218+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:54.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:54.463+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:54.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:54.463+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:54.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:54.463+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:54.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:54.463+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:55.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:54.838+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:55.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:54.838+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:55.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:54.838+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:55.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:54.839+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:55.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:55.418+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:55.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:55.418+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:55.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:55.418+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:55.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:55.419+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:55.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:55.241+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:55.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:55.241+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:55.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:55.241+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:55.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:55.241+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:55.887 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:12:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:12:55.532+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 22209 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:12:56.255 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:55.887+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:56.255 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:55.887+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:56.255 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:55.887+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:56.255 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:55.888+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:56.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:56.255+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:56.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:56.255+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:56.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:56.255+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:56.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:56.256+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:56.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:56.458+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:56.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:56.458+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:56.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:56.458+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:56.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:56.458+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:57.261 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:56.877+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:57.261 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:56.877+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:57.262 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:56.877+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:57.262 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:56.877+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:57.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:57.410+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:57.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:57.410+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:57.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:57.410+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:57.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:57.411+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:57.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:57.260+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:57.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:57.260+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:57.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:57.260+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:57.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:57.262+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:58.280 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:57.828+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:58.280 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:57.828+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:58.280 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:57.828+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:58.280 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:57.829+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:58.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:58.422+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:58.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:58.422+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:58.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:58.422+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:58.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:58.423+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:58.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:58.279+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:58.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:58.279+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:58.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:58.279+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:58.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:58.280+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:59.275 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:58.877+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:59.276 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:58.877+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:59.276 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:58.877+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:12:59.276 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:58.877+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:59.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:59.282+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:59.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:59.282+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:59.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:59.282+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:12:59.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:12:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:12:59.282+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:12:59.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:59.387+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:59.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:59.387+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:59.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:59.387+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:12:59.535 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:12:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:12:59.388+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:00.178 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:59.927+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:00.178 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:59.927+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:00.179 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:59.927+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:00.179 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:12:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:12:59.928+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:00.535 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:00.243+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:00.535 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:00.244+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:00.536 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:00.244+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:00.536 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:00.244+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:00.536 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:00.366+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:00.536 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:00.366+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:00.537 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:00.366+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:00.537 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:00.367+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:00.931 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:13:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:13:00.535+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 22297 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:13:01.256 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:00.931+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:01.256 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:00.931+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:01.256 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:00.931+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:01.256 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:00.932+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:01.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:01.256+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:01.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:01.256+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:01.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:01.256+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:01.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:01.257+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:01.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:01.350+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:01.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:01.350+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:01.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:01.350+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:01.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:01.351+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:02.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:01.979+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:02.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:01.979+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:02.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:01.979+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:02.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:01.980+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:02.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:02.290+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:02.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:02.290+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:02.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:02.290+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:02.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:02.291+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:02.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:02.335+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:02.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:02.335+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:02.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:02.335+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:02.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:02.336+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:03.245 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:02.933+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:03.245 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:02.933+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:03.245 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:02.933+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:03.245 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:02.934+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:03.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:03.330+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:03.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:03.330+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:03.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:03.330+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:03.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:03.331+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:03.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:03.245+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:03.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:03.245+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:03.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:03.245+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:03.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:03.246+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:04.237 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:03.962+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:04.237 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:03.962+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:04.237 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:03.962+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:04.237 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:03.963+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:04.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:04.237+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:04.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:04.237+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:04.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:04.237+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:04.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:04.238+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:04.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:04.368+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:04.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:04.368+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:04.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:04.368+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:04.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:04.368+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:05.223 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:04.972+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:05.223 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:04.972+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:05.223 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:04.972+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:05.224 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:04.973+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:05.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:05.223+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:05.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:05.223+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:05.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:05.223+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:05.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:05.224+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:05.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:05.396+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:05.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:05.396+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:05.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:05.396+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:05.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:05.396+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:05.976 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:13:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:13:05.538+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 22389 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:13:06.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:06.178+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:06.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:06.178+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:06.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:06.178+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:06.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:06.179+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:06.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:05.976+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:06.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:05.976+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:06.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:05.976+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:06.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:05.977+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:06.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:06.421+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:06.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:06.421+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:06.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:06.421+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:06.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:06.422+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:07.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:07.156+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:07.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:07.156+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:07.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:07.156+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:07.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:07.158+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:07.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:07.027+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:07.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:07.027+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:07.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:07.027+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:07.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:07.027+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:07.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:07.470+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:07.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:07.470+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:07.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:07.470+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:07.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:07.471+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:08.444 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:08.074+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:08.445 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:08.074+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:08.445 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:08.074+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:08.445 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:08.075+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:08.445 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:08.138+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:08.445 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:08.138+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:08.445 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:08.138+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:08.446 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:08.138+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:08.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:08.445+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:08.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:08.445+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:08.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:08.445+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:08.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:08.446+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:09.353 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:09.033+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:09.353 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:09.033+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:09.354 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:09.033+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:09.354 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:09.034+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:09.354 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:09.142+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:09.354 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:09.142+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:09.354 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:09.142+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:09.354 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:09.143+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:09.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:09.469+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:09.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:09.469+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:09.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:09.469+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:09.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:09.470+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:10.178 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:09.997+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:10.179 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:09.997+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:10.179 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:09.997+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:10.179 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:09.998+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:10.518 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:10.181+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:10.518 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:10.181+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:10.519 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:10.181+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:10.519 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:10.182+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:10.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:13:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:13:10.541+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 22477 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:13:10.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:10.518+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:10.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:10.518+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:10.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:10.518+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:10.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:10.519+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:11.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:11.155+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:11.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:11.155+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:11.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:11.155+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:11.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:11.156+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:11.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:10.966+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:11.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:10.966+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:11.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:10.966+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:11.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:10.966+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:12.005 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:11.551+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:12.005 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:11.551+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:12.005 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:11.551+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:12.005 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:11.552+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:12.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:12.111+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:12.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:12.111+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:12.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:12.111+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:12.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:12.112+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:12.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:12.005+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:12.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:12.005+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:12.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:12.005+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:12.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:12.006+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:12.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:12.506+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:12.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:12.506+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:12.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:12.506+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:12.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:12.507+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:13.527 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:13.113+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:13.528 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:13.113+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:13.528 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:13.113+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:13.528 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:13.114+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:13.528 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:13.054+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:13.528 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:13.054+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:13.529 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:13.054+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:13.529 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:13.055+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:13.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:13.526+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:13.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:13.526+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:13.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:13.526+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:13.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:13.528+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:14.393 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:14.071+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:14.393 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:14.071+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:14.393 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:14.071+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:14.393 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:14.072+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:14.394 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:14.058+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:14.394 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:14.058+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:14.394 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:14.058+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:14.394 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:14.059+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:14.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:14.563+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:14.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:14.563+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:14.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:14.563+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:14.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:14.564+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:15.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:15.061+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:15.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:15.061+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:15.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:15.061+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:15.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:15.062+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:15.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:15.010+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:15.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:15.010+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:15.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:15.010+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:15.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:15.010+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:15.998 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:13:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:13:15.542+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 22568 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:13:15.999 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:15.561+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:15.999 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:15.561+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:15.999 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:15.561+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:15.999 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:15.561+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:16.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:16.034+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:16.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:16.034+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:16.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:16.034+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:16.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:16.035+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:16.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:15.998+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:16.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:15.998+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:16.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:15.998+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:16.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:15.998+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:16.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:16.515+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:16.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:16.515+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:16.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:16.515+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:16.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:16.516+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:17.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:16.986+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:17.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:16.986+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:17.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:16.986+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:17.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:16.987+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:17.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:16.976+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:17.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:16.976+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:17.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:16.976+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:17.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:16.976+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:17.958 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:17.546+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:17.958 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:17.546+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:17.958 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:17.546+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:17.958 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:17.547+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:18.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:18.003+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:18.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:18.003+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:18.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:18.003+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:18.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:18.004+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:18.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:17.957+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:18.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:17.957+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:18.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:17.957+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:18.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:17.958+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:19.000 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:18.539+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:19.001 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:18.539+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:19.001 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:18.539+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:19.001 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:18.539+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:19.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:19.000+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:19.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:19.000+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:19.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:19.000+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:19.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:19.000+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:19.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:19.000+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:19.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:19.000+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:19.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:19.000+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:19.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:19.001+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:19.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:19.574+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:19.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:19.574+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:19.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:19.574+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:19.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:19.575+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:20.179 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:20.046+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:20.180 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:20.046+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:20.180 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:20.046+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:20.180 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:20.047+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:20.180 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:20.051+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:20.181 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:20.051+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:20.181 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:20.051+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:20.181 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:20.051+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:21.003 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:13:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:13:20.545+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 22656 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:13:21.003 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:20.558+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:21.003 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:20.558+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:21.003 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:20.558+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:21.004 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:20.559+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:21.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:21.002+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:21.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:21.002+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:21.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:21.002+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:21.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:21.003+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:21.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:21.045+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:21.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:21.045+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:21.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:21.045+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:21.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:21.046+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:22.011 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:21.574+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:22.011 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:21.574+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:22.011 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:21.574+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:22.011 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:21.575+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:22.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:22.046+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:22.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:22.046+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:22.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:22.046+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:22.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:22.047+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:22.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:22.010+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:22.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:22.010+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:22.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:22.010+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:22.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:22.011+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:22.985 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:22.565+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:22.986 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:22.565+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:22.986 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:22.565+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:22.986 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:22.566+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:23.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:23.073+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:23.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:23.073+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:23.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:23.073+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:23.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:23.074+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:23.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:22.985+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:23.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:22.985+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:23.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:22.985+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:23.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:22.986+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:24.015 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:23.580+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:24.015 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:23.580+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:24.015 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:23.580+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:24.015 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:23.580+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:24.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:24.065+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:24.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:24.065+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:24.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:24.065+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:24.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:24.065+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:24.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:24.014+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:24.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:24.014+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:24.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:24.014+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:24.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:24.015+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:24.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:24.623+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:24.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:24.623+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:24.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:24.623+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:24.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:24.623+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:25.328 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:25.033+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:25.328 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:25.033+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:25.328 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:25.033+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:25.329 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:25.034+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:25.329 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:25.063+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:25.329 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:25.063+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:25.329 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:25.063+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:25.329 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:25.064+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:25.638 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:13:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:13:25.547+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 22748 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:13:26.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:25.638+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:26.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:25.638+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:26.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:25.638+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:26.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:25.639+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:26.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:26.111+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:26.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:26.111+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:26.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:26.111+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:26.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:26.112+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:26.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:26.066+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:26.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:26.066+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:26.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:26.066+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:26.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:26.067+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:27.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:26.657+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:27.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:26.657+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:27.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:26.657+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:27.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:26.658+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:27.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:27.068+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:27.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:27.068+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:27.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:27.068+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:27.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:27.069+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:27.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:27.131+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:27.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:27.132+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:27.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:27.132+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:27.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:27.132+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:28.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:27.688+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:28.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:27.688+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:28.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:27.688+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:28.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:27.689+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:28.089+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:28.089+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:28.089+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:28.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:28.089+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:28.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:28.112+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:28.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:28.112+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:28.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:28.112+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:28.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:28.113+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:29.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:28.709+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:29.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:28.709+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:29.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:28.709+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:29.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:28.710+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:29.511 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:29.123+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:29.511 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:29.123+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:29.511 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:29.123+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:29.512 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:29.124+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:29.512 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:29.100+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:29.512 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:29.100+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:29.512 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:29.100+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:29.512 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:29.100+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:29.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:29.747+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:29.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:29.747+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:29.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:29.747+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:29.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:29.747+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:30.106 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:30.083+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:30.106 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:30.083+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:30.106 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:30.083+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:30.106 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:30.084+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:30.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:30.105+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:30.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:30.105+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:30.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:30.105+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:30.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:30.106+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:31.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:13:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:13:30.550+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 22836 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:13:31.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:30.702+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:31.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:30.702+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:31.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:30.702+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:31.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:30.703+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:31.472 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:31.111+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:31.473 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:31.111+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:31.473 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:31.111+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:31.473 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:31.112+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:31.473 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:31.132+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:31.473 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:31.132+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:31.474 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:31.132+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:31.474 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:31.133+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:32.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:31.727+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:32.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:31.727+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:32.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:31.727+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:32.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:31.728+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:32.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:32.128+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:32.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:32.128+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:32.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:32.128+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:32.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:32.129+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:32.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:32.178+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:32.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:32.178+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:32.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:32.178+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:32.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:32.179+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:33.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:32.731+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:33.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:32.731+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:33.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:32.731+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:33.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:32.732+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:33.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:33.134+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:33.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:33.134+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:33.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:33.134+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:33.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:33.135+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:33.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:33.152+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:33.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:33.152+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:33.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:33.152+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:33.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:33.153+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:34.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:33.764+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:34.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:33.764+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:34.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:33.764+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:34.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:33.764+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:34.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:34.168+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:34.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:34.168+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:34.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:34.168+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:34.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:34.169+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:34.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:34.136+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:34.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:34.136+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:34.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:34.136+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:34.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:34.137+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:35.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:34.721+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:35.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:34.721+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:35.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:34.721+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:35.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:34.722+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:35.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:35.154+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:35.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:35.154+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:35.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:35.154+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:35.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:35.154+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:35.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:35.097+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:35.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:35.097+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:35.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:35.097+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:35.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:35.097+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:36.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:13:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:13:35.552+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 22927 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:13:36.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:35.692+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:36.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:35.692+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:36.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:35.692+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:36.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:35.692+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:36.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:36.112+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:36.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:36.112+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:36.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:36.112+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:36.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:36.112+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:36.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:36.055+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:36.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:36.055+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:36.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:36.055+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:36.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:36.056+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:37.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:36.705+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:37.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:36.705+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:37.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:36.705+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:37.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:36.706+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:37.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:37.083+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:37.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:37.083+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:37.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:37.083+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:37.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:37.083+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:37.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:37.074+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:37.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:37.074+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:37.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:37.074+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:37.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:37.075+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:38.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:37.716+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:38.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:37.716+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:38.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:37.716+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:38.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:37.717+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:38.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:38.122+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:38.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:38.122+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:38.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:38.122+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:38.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:38.122+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:38.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:38.122+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:38.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:38.122+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:38.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:38.122+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:38.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:38.123+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:38.742+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:38.742+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:38.742+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:39.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:38.744+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:39.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:39.124+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:39.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:39.124+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:39.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:39.124+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:39.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:39.124+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:39.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:39.172+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:39.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:39.172+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:39.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:39.172+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:39.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:39.173+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:39.853 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:39.769+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:39.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:39.769+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:39.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:39.769+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:39.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:39.769+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:40.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:40.108+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:40.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:40.108+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:40.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:40.108+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:40.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:40.108+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:40.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:40.133+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:40.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:40.133+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:40.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:40.133+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:40.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:40.134+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:41.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:13:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:13:40.554+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 23015 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:13:41.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:40.789+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:41.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:40.789+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:41.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:40.789+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:41.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:40.790+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:41.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:41.072+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:41.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:41.072+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:41.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:41.072+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:41.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:41.073+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:41.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:41.098+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:41.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:41.098+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:41.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:41.098+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:41.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:41.099+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:42.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:42.078+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:42.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:42.078+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:42.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:42.078+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:42.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:42.079+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:42.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:42.051+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:42.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:42.052+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:42.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:42.052+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:42.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:42.052+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:42.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:41.837+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:42.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:41.837+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:42.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:41.837+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:42.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:41.838+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:43.060 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:43.049+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:43.060 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:43.049+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:43.060 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:43.049+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:43.061 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:43.049+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:43.061 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:42.801+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:43.061 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:42.801+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:43.061 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:42.801+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:43.061 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:42.801+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:43.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:43.060+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:43.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:43.060+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:43.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:43.060+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:43.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:43.061+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:44.031 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:43.789+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:44.031 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:43.789+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:44.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:43.789+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:44.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:43.790+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:44.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:44.091+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:44.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:44.091+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:44.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:44.091+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:44.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:44.092+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:44.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:44.031+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:44.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:44.031+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:44.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:44.031+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:44.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:44.032+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:45.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:44.797+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:45.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:44.797+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:45.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:44.797+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:45.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:44.798+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:45.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:45.063+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:45.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:45.063+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:45.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:45.063+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:45.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:45.064+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:45.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:45.066+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:45.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:45.066+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:45.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:45.066+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:45.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:45.066+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:46.030 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:13:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:13:45.557+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 23107 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:13:46.030 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:45.788+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:46.030 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:45.788+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:46.031 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:45.788+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:46.031 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:45.789+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:46.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:46.029+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:46.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:46.029+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:46.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:46.029+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:46.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:46.030+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:46.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:46.054+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:46.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:46.054+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:46.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:46.054+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:46.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:46.055+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:47.015 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:46.746+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:47.015 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:46.746+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:47.015 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:46.746+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:47.015 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:46.747+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:47.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:47.014+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:47.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:47.014+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:47.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:47.014+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:47.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:47.015+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:47.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:47.063+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:47.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:47.063+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:47.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:47.063+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:47.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:47.064+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:48.067 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:48.029+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:48.067 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:48.029+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:48.068 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:48.029+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:48.068 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:48.030+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:48.068 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:47.786+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:48.068 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:47.786+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:48.068 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:47.786+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:48.069 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:47.787+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:48.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:48.066+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:48.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:48.066+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:48.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:48.066+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:48.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:48.067+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:49.024 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:48.739+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:49.024 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:48.739+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:49.025 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:48.739+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:49.025 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:48.740+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:49.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:49.117+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:49.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:49.117+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:49.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:49.117+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:49.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:49.118+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:49.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:49.024+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:49.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:49.024+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:49.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:49.024+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:49.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:49.024+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:50.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:49.778+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:50.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:49.778+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:50.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:49.778+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:50.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:49.779+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:50.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:50.043+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:50.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:50.043+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:50.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:50.043+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:50.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:50.044+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:50.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:50.094+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:50.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:50.094+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:50.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:50.094+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:50.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:50.096+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:51.001 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:13:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:13:50.559+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 23195 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:13:51.002 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:50.768+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:51.002 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:50.768+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:51.002 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:50.768+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:51.002 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:50.769+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:51.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:51.124+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:51.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:51.124+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:51.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:51.124+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:51.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:51.124+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:51.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:51.001+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:51.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:51.001+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:51.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:51.001+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:51.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:51.002+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:52.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:51.762+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:52.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:51.762+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:52.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:51.762+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:52.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:51.762+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:52.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:52.000+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:52.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:52.000+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:52.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:52.000+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:52.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:52.001+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:52.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:52.171+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:52.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:52.171+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:52.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:52.171+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:52.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:52.172+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:52.977 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:52.717+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:52.977 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:52.717+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:52.977 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:52.717+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:52.977 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:52.717+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:53.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:52.985+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:53.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:52.985+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:53.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:52.985+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:53.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:52.986+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:53.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:53.167+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:53.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:53.167+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:53.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:53.167+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:53.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:53.168+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:53.954 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:53.730+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:53.955 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:53.730+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:53.955 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:53.730+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:53.955 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:53.730+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:54.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:54.144+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:54.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:54.144+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:54.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:54.144+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:54.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:54.145+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:54.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:53.954+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:54.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:53.954+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:54.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:53.954+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:54.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:53.955+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:55.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:54.704+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:55.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:54.704+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:55.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:54.704+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:55.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:54.705+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:55.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:54.935+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:55.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:54.935+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:55.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:54.935+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:55.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:54.935+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:55.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:55.181+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:55.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:55.181+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:55.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:55.181+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:55.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:55.182+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:55.914 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:13:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:13:55.562+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 23286 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:13:55.914 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:55.707+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:55.914 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:55.707+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:55.914 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:55.707+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:55.915 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:55.708+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:56.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:55.913+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:56.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:55.913+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:56.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:55.913+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:56.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:55.914+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:56.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:56.157+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:56.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:56.157+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:56.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:56.157+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:56.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:56.158+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:57.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:56.900+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:57.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:56.900+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:57.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:56.900+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:57.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:56.901+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:57.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:56.675+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:57.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:56.675+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:57.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:56.675+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:57.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:56.676+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:57.510 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:57.136+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:57.510 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:57.136+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:57.510 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:57.136+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:57.510 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:57.136+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:57.771 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:57.677+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:57.771 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:57.677+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:57.771 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:57.677+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:57.771 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:57.678+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:58.022 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:57.909+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:58.022 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:57.909+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:58.023 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:57.909+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:58.023 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:57.910+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:58.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:58.144+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:58.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:58.144+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:58.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:58.144+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:58.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:58.146+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:59.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:58.903+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:59.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:58.903+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:59.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:58.903+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:13:59.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:58.905+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:59.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:58.678+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:59.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:58.678+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:59.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:58.678+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:59.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:58.679+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:59.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:59.167+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:59.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:59.167+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:59.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:59.167+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:13:59.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:13:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:13:59.168+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:13:59.923 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:59.630+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:59.923 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:59.630+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:59.923 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:59.630+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:13:59.923 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:13:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:13:59.631+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:00.182 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:59.922+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:00.182 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:59.922+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:00.182 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:59.922+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:00.182 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:13:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:13:59.923+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:00.183 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:00.156+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:00.183 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:00.156+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:00.183 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:00.156+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:00.183 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:00.156+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:00.910 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:14:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:14:00.565+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 23374 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:14:00.910 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:00.613+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:00.910 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:00.613+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:00.910 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:00.613+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:00.910 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:00.614+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:01.174 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:00.909+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:01.175 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:00.909+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:01.175 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:00.909+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:01.175 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:00.910+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:01.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:01.174+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:01.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:01.174+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:01.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:01.174+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:01.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:01.175+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:01.943 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:01.568+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:01.943 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:01.568+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:01.943 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:01.568+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:01.943 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:01.569+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:02.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:01.942+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:02.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:01.942+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:02.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:01.942+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:02.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:01.943+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:02.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:02.144+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:02.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:02.144+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:02.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:02.144+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:02.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:02.145+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:02.915 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:02.544+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:02.915 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:02.544+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:02.915 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:02.544+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:02.915 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:02.545+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:03.176 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:02.914+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:03.176 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:02.914+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:03.176 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:02.914+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:03.176 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:02.915+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:03.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:03.175+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:03.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:03.175+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:03.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:03.175+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:03.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:03.176+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:03.884 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:03.541+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:03.884 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:03.541+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:03.885 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:03.541+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:03.885 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:03.542+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:04.218 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:03.884+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:04.219 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:03.884+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:04.219 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:03.884+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:04.219 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:03.885+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:04.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:04.218+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:04.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:04.218+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:04.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:04.218+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:04.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:04.219+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:04.934 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:04.569+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:04.935 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:04.569+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:04.935 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:04.569+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:04.935 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:04.570+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:05.243 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:04.934+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:05.244 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:04.934+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:05.244 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:04.934+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:05.244 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:04.935+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:05.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:05.243+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:05.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:05.243+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:05.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:05.243+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:05.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:05.244+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:05.933 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:14:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:14:05.567+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 23466 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:14:05.933 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:05.540+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:05.934 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:05.540+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:05.934 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:05.540+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:05.934 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:05.541+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:06.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:05.933+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:06.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:05.933+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:06.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:05.933+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:06.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:05.934+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:06.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:06.522+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:06.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:06.522+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:06.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:06.522+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:06.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:06.523+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:06.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:06.286+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:06.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:06.286+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:06.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:06.286+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:06.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:06.287+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:07.275 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:06.927+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:07.275 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:06.927+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:07.275 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:06.927+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:07.275 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:06.927+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:07.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:07.522+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:07.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:07.522+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:07.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:07.522+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:07.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:07.523+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:07.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:07.274+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:07.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:07.274+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:07.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:07.274+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:07.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:07.275+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:08.241 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:07.910+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:08.241 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:07.910+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:08.241 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:07.910+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:08.241 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:07.911+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:08.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:08.479+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:08.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:08.479+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:08.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:08.479+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:08.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:08.480+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:08.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:08.240+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:08.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:08.240+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:08.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:08.240+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:08.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:08.241+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:09.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:08.881+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:09.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:08.881+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:09.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:08.881+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:09.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:08.882+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:09.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:09.484+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:09.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:09.484+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:09.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:09.484+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:09.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:09.484+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:09.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:09.273+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:09.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:09.273+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:09.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:09.273+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:09.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:09.274+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:10.183 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:09.921+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:10.183 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:09.921+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:10.183 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:09.921+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:10.184 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:09.922+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:10.483 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:10.245+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:10.484 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:10.245+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:10.484 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:10.245+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:10.484 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:10.245+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:10.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:14:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:14:10.570+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 23554 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:14:10.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:10.483+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:10.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:10.483+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:10.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:10.483+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:10.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:10.484+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:11.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:10.940+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:11.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:10.940+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:11.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:10.940+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:11.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:10.940+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:11.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:11.439+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:11.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:11.439+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:11.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:11.439+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:11.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:11.439+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:11.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:11.285+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:11.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:11.285+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:11.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:11.285+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:11.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:11.285+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:12.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:11.903+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:12.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:11.903+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:12.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:11.903+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:12.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:11.904+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:12.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:12.293+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:12.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:12.293+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:12.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:12.293+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:12.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:12.294+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:12.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:12.450+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:12.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:12.450+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:12.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:12.450+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:12.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:12.451+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:13.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:12.866+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:13.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:12.866+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:13.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:12.866+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:13.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:12.867+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:13.611 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:13.493+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:13.611 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:13.493+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:13.611 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:13.493+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:13.612 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:13.495+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:13.612 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:13.314+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:13.612 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:13.314+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:13.612 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:13.314+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:13.612 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:13.315+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:14.276 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:13.908+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:14.276 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:13.908+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:14.276 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:13.908+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:14.277 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:13.909+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:14.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:14.275+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:14.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:14.275+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:14.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:14.275+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:14.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:14.276+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:14.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:14.505+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:14.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:14.505+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:14.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:14.505+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:14.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:14.506+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:15.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:14.859+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:15.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:14.859+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:15.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:14.859+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:15.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:14.860+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:15.573 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:15.521+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:15.574 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:15.521+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:15.574 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:15.521+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:15.574 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:15.522+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:15.574 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:15.297+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:15.575 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:15.297+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:15.575 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:15.297+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:15.575 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:15.297+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:15.903 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:14:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:14:15.573+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 23645 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:14:16.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:15.903+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:16.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:15.903+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:16.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:15.903+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:16.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:15.904+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:16.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:16.497+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:16.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:16.497+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:16.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:16.497+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:16.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:16.497+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:16.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:16.282+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:16.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:16.282+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:16.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:16.282+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:16.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:16.283+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:17.238 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:16.895+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:17.238 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:16.895+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:17.238 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:16.895+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:17.238 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:16.896+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:17.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:17.237+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:17.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:17.237+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:17.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:17.237+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:17.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:17.238+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:17.942 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:17.536+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:17.942 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:17.536+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:17.942 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:17.536+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:17.943 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:17.537+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:18.237 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:17.942+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:18.237 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:17.942+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:18.237 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:17.942+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:18.238 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:17.942+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:18.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:18.236+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:18.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:18.236+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:18.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:18.237+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:18.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:18.237+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:18.986 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:18.538+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:18.986 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:18.538+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:18.987 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:18.538+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:18.987 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:18.538+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:19.237 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:18.986+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:19.237 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:18.986+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:19.237 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:18.986+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:19.237 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:18.986+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:19.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:19.236+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:19.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:19.236+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:19.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:19.236+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:19.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:19.237+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:19.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:19.537+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:19.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:19.537+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:19.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:19.537+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:19.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:19.538+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:20.183 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:19.943+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:20.183 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:19.943+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:20.183 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:19.943+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:20.184 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:19.944+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:20.511 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:20.281+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:20.511 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:20.281+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:20.511 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:20.281+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:20.511 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:20.281+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:20.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:14:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:14:20.575+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 23733 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:14:20.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:20.510+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:20.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:20.510+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:20.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:20.510+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:20.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:20.511+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:21.264 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:20.990+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:21.264 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:20.990+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:21.264 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:20.990+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:21.264 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:20.990+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:21.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:21.483+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:21.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:21.483+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:21.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:21.483+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:21.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:21.483+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:21.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:21.264+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:21.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:21.264+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:21.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:21.264+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:21.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:21.265+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:22.278 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:22.028+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:22.278 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:22.028+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:22.279 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:22.028+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:22.279 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:22.028+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:22.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:22.444+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:22.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:22.444+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:22.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:22.444+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:22.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:22.445+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:22.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:22.278+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:22.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:22.278+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:22.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:22.278+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:22.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:22.279+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:23.398 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:23.051+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:23.399 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:23.051+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:23.399 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:23.051+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:23.399 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:23.052+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:23.399 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:23.238+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:23.399 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:23.238+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:23.400 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:23.238+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:23.400 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:23.239+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:23.697 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:23.398+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:23.697 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:23.398+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:23.697 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:23.398+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:23.698 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:23.399+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:24.407 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:24.039+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:24.407 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:24.039+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:24.408 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:24.039+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:24.408 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:24.040+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:24.408 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:24.205+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:24.409 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:24.205+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:24.409 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:24.205+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:24.409 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:24.206+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:24.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:24.407+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:24.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:24.407+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:24.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:24.407+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:24.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:24.408+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:25.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:25.044+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:25.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:25.044+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:25.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:25.044+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:25.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:25.045+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:25.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:25.157+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:25.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:25.157+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:25.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:25.157+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:25.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:25.158+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:25.702 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:14:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:14:25.579+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 23834 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:14:25.702 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:25.425+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:25.702 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:25.425+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:25.702 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:25.425+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:25.702 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:25.426+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:26.471 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:26.087+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:26.472 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:26.087+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:26.472 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:26.087+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:26.472 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:26.088+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:26.473 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:26.199+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:26.473 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:26.199+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:26.473 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:26.199+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:26.473 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:26.200+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:26.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:26.472+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:26.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:26.472+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:26.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:26.472+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:26.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:26.473+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:27.508 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:27.098+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:27.508 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:27.098+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:27.508 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:27.098+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:27.508 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:27.099+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:27.508 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:27.164+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:27.509 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:27.164+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:27.509 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:27.164+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:27.509 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:27.165+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:27.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:27.508+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:27.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:27.508+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:27.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:27.508+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:27.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:27.510+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:28.511 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:28.126+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:28.512 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:28.126+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:28.512 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:28.126+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:28.512 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:28.127+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:28.512 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:28.074+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:28.513 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:28.074+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:28.513 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:28.074+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:28.513 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:28.074+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:28.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:28.512+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:28.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:28.512+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:28.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:28.512+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:28.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:28.513+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:29.470 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:29.135+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:29.471 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:29.135+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:29.471 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:29.135+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:29.471 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:29.136+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:29.472 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:29.050+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:29.472 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:29.050+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:29.472 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:29.050+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:29.472 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:29.050+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:29.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:29.470+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:29.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:29.470+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:29.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:29.470+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:29.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:29.472+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:30.178 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:30.075+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:30.178 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:30.075+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:30.178 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:30.075+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:30.178 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:30.076+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:30.451 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:30.178+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:30.451 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:30.178+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:30.451 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:30.178+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:30.451 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:30.179+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:30.706 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:14:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:14:30.582+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 23922 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:14:30.707 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:30.451+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:30.707 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:30.451+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:30.707 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:30.451+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:30.707 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:30.452+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:31.425 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:31.099+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:31.426 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:31.099+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:31.426 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:31.099+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:31.426 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:31.100+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:31.426 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:31.185+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:31.427 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:31.185+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:31.427 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:31.185+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:31.427 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:31.185+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:31.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:31.426+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:31.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:31.426+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:31.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:31.426+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:31.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:31.427+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:32.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:32.180+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:32.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:32.180+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:32.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:32.180+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:32.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:32.181+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:32.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:32.381+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:32.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:32.381+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:32.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:32.381+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:32.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:32.381+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:32.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:32.147+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:32.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:32.147+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:32.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:32.147+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:32.535 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:32.147+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:33.170 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:33.134+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:33.171 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:33.134+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:33.171 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:33.134+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:33.171 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:33.134+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:33.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:33.171+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:33.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:33.171+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:33.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:33.171+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:33.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:33.172+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:33.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:33.409+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:33.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:33.409+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:33.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:33.409+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:33.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:33.410+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:34.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:34.128+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:34.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:34.128+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:34.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:34.128+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:34.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:34.129+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:34.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:34.133+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:34.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:34.133+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:34.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:34.133+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:34.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:34.133+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:34.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:34.362+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:34.535 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:34.362+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:34.535 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:34.362+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:34.535 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:34.364+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:35.387 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:35.096+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:35.388 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:35.096+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:35.388 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:35.096+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:35.388 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:35.097+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:35.388 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:35.093+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:35.389 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:35.093+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:35.389 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:35.093+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:35.389 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:35.094+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:35.708 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:14:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:14:35.583+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 24013 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:14:35.708 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:35.387+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:35.708 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:35.387+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:35.708 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:35.387+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:35.708 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:35.387+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:36.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:36.097+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:36.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:36.097+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:36.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:36.097+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:36.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:36.098+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:36.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:36.072+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:36.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:36.072+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:36.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:36.072+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:36.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:36.073+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:36.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:36.428+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:36.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:36.428+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:36.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:36.428+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:36.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:36.429+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:37.421 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:37.077+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:37.421 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:37.077+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:37.421 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:37.077+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:37.421 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:37.078+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:37.422 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:37.042+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:37.422 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:37.042+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:37.422 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:37.042+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:37.422 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:37.043+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:37.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:37.420+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:37.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:37.420+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:37.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:37.420+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:37.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:37.421+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:38.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:38.017+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:38.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:38.017+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:38.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:38.017+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:38.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:38.018+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:38.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:38.064+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:38.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:38.064+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:38.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:38.064+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:38.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:38.065+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:38.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:38.373+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:38.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:38.373+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:38.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:38.373+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:38.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:38.374+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:39.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:38.987+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:39.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:38.987+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:39.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:38.987+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:39.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:38.988+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:39.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:39.062+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:39.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:39.062+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:39.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:39.062+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:39.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:39.063+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:39.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:39.325+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:39.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:39.325+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:39.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:39.325+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:39.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:39.326+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:40.184 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:40.051+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:40.184 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:40.051+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:40.184 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:40.051+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:40.185 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:40.051+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:40.185 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:40.006+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:40.185 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:40.006+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:40.185 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:40.006+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:40.185 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:40.007+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:40.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:40.328+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:40.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:40.328+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:40.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:40.328+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:40.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:40.328+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:40.993 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:14:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:14:40.586+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 24100 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:14:41.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:41.095+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:41.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:41.095+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:41.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:41.095+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:41.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:41.096+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:41.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:40.993+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:41.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:40.993+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:41.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:40.993+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:41.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:40.994+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:41.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:41.378+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:41.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:41.378+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:41.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:41.378+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:41.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:41.379+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:42.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:42.115+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:42.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:42.115+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:42.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:42.115+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:42.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:42.116+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:42.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:41.990+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:42.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:41.990+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:42.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:41.990+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:42.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:41.991+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:42.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:42.407+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:42.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:42.407+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:42.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:42.407+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:42.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:42.408+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:43.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:43.158+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:43.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:43.158+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:43.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:43.158+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:43.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:43.159+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:43.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:43.008+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:43.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:43.008+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:43.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:43.008+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:43.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:43.009+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:43.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:43.396+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:43.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:43.396+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:43.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:43.396+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:43.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:43.396+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:44.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:44.136+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:44.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:44.136+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:44.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:44.136+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:44.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:44.136+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:44.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:43.979+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:44.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:43.979+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:44.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:43.979+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:44.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:43.979+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:44.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:44.352+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:44.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:44.352+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:44.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:44.352+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:44.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:44.353+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:45.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:44.953+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:45.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:44.953+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:45.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:44.953+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:45.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:44.953+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:45.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:45.174+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:45.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:45.174+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:45.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:45.174+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:45.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:45.174+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:45.589 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:45.310+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:45.589 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:45.310+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:45.590 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:45.310+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:45.590 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:45.312+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:45.945 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:14:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:14:45.589+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 24192 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:14:46.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:45.944+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:46.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:45.944+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:46.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:45.944+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:46.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:45.945+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:46.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:46.179+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:46.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:46.179+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:46.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:46.179+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:46.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:46.180+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:46.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:46.312+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:46.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:46.312+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:46.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:46.312+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:46.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:46.313+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:47.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:46.928+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:47.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:46.928+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:47.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:46.928+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:47.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:46.929+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:47.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:47.138+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:47.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:47.138+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:47.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:47.138+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:47.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:47.139+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:47.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:47.305+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:47.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:47.305+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:47.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:47.305+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:47.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:47.306+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:48.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:48.126+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:48.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:48.126+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:48.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:48.126+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:48.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:48.127+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:48.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:47.960+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:48.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:47.960+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:48.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:47.960+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:48.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:47.961+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:48.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:48.333+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:48.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:48.333+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:48.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:48.333+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:48.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:48.335+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:49.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:49.174+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:49.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:49.174+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:49.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:49.174+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:49.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:49.175+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:49.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:48.968+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:49.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:48.968+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:49.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:48.968+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:49.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:48.969+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:49.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:49.374+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:49.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:49.374+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:49.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:49.374+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:49.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:49.374+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:50.172 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:49.999+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:50.172 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:49.999+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:50.172 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:49.999+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:50.172 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:50.000+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:50.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:50.331+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:50.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:50.331+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:50.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:50.331+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:50.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:50.332+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:50.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:50.196+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:50.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:50.196+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:50.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:50.196+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:50.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:50.196+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:51.003 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:14:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:14:50.591+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 24279 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:14:51.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:51.203+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:51.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:51.203+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:51.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:51.203+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:51.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:51.204+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:51.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:51.003+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:51.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:51.003+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:51.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:51.003+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:51.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:51.004+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:51.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:51.363+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:51.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:51.363+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:51.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:51.363+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:51.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:51.364+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:52.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:51.978+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:52.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:51.978+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:52.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:51.978+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:52.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:51.979+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:52.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:52.190+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:52.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:52.190+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:52.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:52.190+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:52.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:52.190+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:52.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:52.323+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:52.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:52.323+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:52.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:52.323+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:52.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:52.324+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:53.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:53.165+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:53.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:53.165+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:53.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:53.165+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:53.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:53.165+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:53.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:52.970+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:53.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:52.970+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:53.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:52.970+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:53.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:52.971+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:53.613 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:53.308+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:53.613 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:53.308+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:53.613 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:53.308+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:53.613 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:53.309+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:54.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:54.160+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:54.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:54.160+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:54.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:54.160+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:54.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:54.161+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:54.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:53.970+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:54.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:53.970+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:54.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:53.970+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:54.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:53.971+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:54.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:54.359+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:54.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:54.359+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:54.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:54.359+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:54.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:54.359+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:55.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:55.186+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:55.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:55.186+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:55.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:55.186+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:55.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:55.187+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:55.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:54.993+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:55.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:54.993+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:55.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:54.993+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:55.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:54.994+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:55.595 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:55.317+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:55.595 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:55.317+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:55.595 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:55.317+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:55.595 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:55.318+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:56.011 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:14:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:14:55.594+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 24369 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:14:56.271 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:56.010+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:56.271 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:56.010+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:56.271 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:56.010+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:56.271 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:56.011+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:56.271 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:56.182+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:56.272 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:56.182+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:56.272 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:56.182+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:56.272 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:56.183+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:56.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:56.270+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:56.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:56.270+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:56.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:56.270+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:56.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:56.271+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:57.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:57.154+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:57.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:57.154+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:57.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:57.154+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:57.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:57.155+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:57.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:57.277+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:57.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:57.277+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:57.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:57.277+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:57.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:57.278+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:57.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:57.033+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:57.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:57.033+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:57.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:57.033+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:57.535 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:57.033+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:58.257 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:58.131+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:58.258 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:58.131+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:58.258 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:58.131+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:58.258 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:58.132+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:58.258 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:57.996+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:58.259 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:57.996+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:58.259 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:57.996+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:58.259 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:57.997+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:58.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:58.256+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:58.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:58.256+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:58.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:58.256+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:58.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:58.257+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:59.281 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:59.095+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:59.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:59.095+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:59.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:59.095+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:14:59.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:14:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:14:59.096+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:59.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:59.002+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:59.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:59.002+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:59.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:59.002+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:14:59.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:59.003+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:14:59.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:59.281+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:59.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:59.281+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:59.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:59.281+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:14:59.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:14:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:14:59.281+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:00.185 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:00.069+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:00.185 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:00.069+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:00.186 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:00.069+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:00.186 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:00.070+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:00.186 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:59.957+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:00.186 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:59.957+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:00.187 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:59.957+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:00.187 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:14:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:14:59.958+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:00.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:00.296+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:00.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:00.296+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:00.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:00.296+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:00.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:00.297+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:00.912 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:15:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:15:00.597+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 24456 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:15:01.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:01.098+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:01.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:01.098+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:01.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:01.098+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:01.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:01.099+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:01.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:00.911+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:01.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:00.911+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:01.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:00.911+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:01.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:00.912+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:01.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:01.327+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:01.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:01.327+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:01.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:01.327+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:01.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:01.328+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:02.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:02.100+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:02.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:02.100+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:02.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:02.100+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:02.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:02.101+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:02.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:01.873+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:02.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:01.873+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:02.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:01.873+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:02.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:01.874+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:02.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:02.302+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:02.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:02.302+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:02.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:02.302+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:02.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:02.303+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:03.135 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:02.840+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:03.135 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:02.840+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:03.135 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:02.840+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:03.135 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:02.841+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:03.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:03.134+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:03.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:03.134+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:03.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:03.134+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:03.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:03.135+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:03.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:03.286+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:03.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:03.286+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:03.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:03.286+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:03.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:03.287+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:04.268 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:03.867+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:04.268 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:03.867+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:04.268 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:03.867+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:04.268 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:03.868+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:04.268 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:04.090+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:04.269 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:04.090+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:04.269 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:04.090+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:04.269 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:04.091+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:04.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:04.267+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:04.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:04.267+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:04.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:04.267+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:04.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:04.268+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:05.125 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:04.850+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:05.125 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:04.850+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:05.125 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:04.850+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:05.126 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:04.851+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:05.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:05.124+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:05.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:05.124+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:05.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:05.124+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:05.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:05.125+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:05.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:05.254+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:05.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:05.254+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:05.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:05.254+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:05.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:05.254+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:06.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:15:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:15:05.600+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 24546 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:15:06.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:05.812+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:06.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:05.812+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:06.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:05.812+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:06.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:05.812+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:06.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:06.111+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:06.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:06.111+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:06.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:06.111+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:06.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:06.111+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:06.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:06.282+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:06.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:06.282+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:06.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:06.282+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:06.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:06.283+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:07.159 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:06.840+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:07.159 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:06.840+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:07.159 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:06.840+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:07.160 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:06.841+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:07.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:07.158+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:07.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:07.158+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:07.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:07.158+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:07.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:07.159+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:07.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:07.316+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:07.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:07.316+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:07.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:07.316+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:07.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:07.316+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:08.123 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:07.791+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:08.124 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:07.791+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:08.124 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:07.791+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:08.124 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:07.792+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:08.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:08.328+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:08.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:08.328+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:08.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:08.328+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:08.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:08.329+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:08.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:08.123+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:08.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:08.123+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:08.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:08.123+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:08.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:08.123+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:09.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:08.757+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:09.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:08.757+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:09.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:08.757+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:09.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:08.758+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:09.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:09.342+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:09.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:09.342+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:09.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:09.342+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:09.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:09.342+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:09.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:09.153+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:09.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:09.153+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:09.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:09.153+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:09.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:09.153+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:10.105 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:09.784+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:10.106 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:09.784+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:10.106 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:09.784+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:10.106 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:09.785+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:10.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:10.105+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:10.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:10.105+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:10.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:10.105+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:10.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:10.105+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:10.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:10.296+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:10.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:10.296+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:10.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:10.296+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:10.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:10.297+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:11.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:15:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:15:10.603+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 24635 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:15:11.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:10.810+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:11.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:10.810+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:11.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:10.810+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:11.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:10.811+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:11.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:11.109+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:11.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:11.109+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:11.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:11.109+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:11.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:11.109+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:11.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:11.314+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:11.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:11.314+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:11.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:11.314+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:11.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:11.314+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:12.092 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:11.812+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:12.093 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:11.812+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:12.093 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:11.812+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:12.093 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:11.813+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:12.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:12.290+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:12.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:12.290+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:12.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:12.290+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:12.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:12.291+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:12.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:12.092+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:12.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:12.092+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:12.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:12.092+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:12.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:12.093+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:13.143 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:12.857+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:13.144 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:12.857+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:13.144 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:12.857+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:13.144 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:12.857+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:13.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:13.143+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:13.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:13.143+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:13.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:13.143+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:13.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:13.143+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:13.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:13.306+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:13.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:13.306+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:13.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:13.306+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:13.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:13.307+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:14.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:13.813+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:14.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:13.813+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:14.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:13.813+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:14.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:13.814+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:14.352 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:14.351+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:14.353 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:14.351+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:14.353 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:14.351+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:14.353 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:14.102+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:14.353 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:14.102+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:14.353 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:14.102+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:14.354 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:14.102+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:14.765 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:14.353+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:15.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:14.765+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:15.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:14.765+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:15.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:14.765+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:15.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:14.765+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:15.327 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:15.077+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:15.327 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:15.077+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:15.327 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:15.077+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:15.327 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:15.078+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:15.606 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:15.326+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:15.606 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:15.326+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:15.606 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:15.326+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:15.607 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:15.327+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:15.921 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:15:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:15:15.605+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 24724 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:15:15.922 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:15.799+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:15.922 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:15.799+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:15.922 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:15.799+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:15.922 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:15.799+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:16.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:16.072+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:16.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:16.072+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:16.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:16.072+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:16.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:16.073+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:16.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:16.290+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:16.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:16.290+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:16.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:16.290+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:16.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:16.291+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:17.111 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:16.790+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:17.111 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:16.790+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:17.111 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:16.790+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:17.111 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:16.790+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:17.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:17.110+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:17.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:17.110+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:17.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:17.110+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:17.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:17.111+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:17.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:17.251+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:17.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:17.251+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:17.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:17.251+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:17.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:17.252+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:18.234 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:18.061+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:18.235 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:18.061+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:18.235 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:18.061+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:18.235 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:18.062+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:18.236 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:17.833+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:18.236 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:17.833+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:18.236 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:17.833+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:18.236 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:17.833+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:18.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:18.234+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:18.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:18.234+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:18.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:18.234+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:18.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:18.234+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:19.210 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:19.035+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:19.211 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:19.035+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:19.211 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:19.035+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:19.211 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:19.036+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:19.211 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:18.829+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:19.211 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:18.829+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:19.212 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:18.829+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:19.212 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:18.830+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:19.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:19.209+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:19.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:19.209+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:19.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:19.209+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:19.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:19.210+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:20.164 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:20.035+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:20.165 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:20.035+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:20.165 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:20.035+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:20.165 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:20.035+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:20.165 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:19.863+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:20.165 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:19.863+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:20.165 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:19.863+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:20.166 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:19.864+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:20.415 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:20.163+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:20.415 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:20.163+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:20.415 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:20.163+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:20.416 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:20.165+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:20.735 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:15:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:15:20.608+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 24813 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:15:21.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:20.893+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:21.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:20.893+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:21.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:20.893+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:21.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:20.894+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:21.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:21.084+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:21.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:21.084+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:21.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:21.084+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:21.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:21.085+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:21.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:21.165+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:21.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:21.165+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:21.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:21.165+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:21.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:21.165+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:22.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:21.916+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:22.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:21.916+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:22.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:21.916+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:22.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:21.917+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:22.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:22.133+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:22.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:22.133+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:22.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:22.133+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:22.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:22.134+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:22.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:22.111+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:22.285 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:22.111+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:22.285 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:22.111+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:22.285 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:22.112+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:23.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:22.871+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:23.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:22.871+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:23.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:22.871+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:23.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:22.872+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:23.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:23.120+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:23.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:23.120+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:23.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:23.120+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:23.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:23.121+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:23.285 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:23.065+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:23.285 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:23.065+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:23.285 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:23.065+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:23.285 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:23.066+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:24.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:23.854+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:24.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:23.854+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:24.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:23.854+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:24.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:23.855+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:24.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:24.060+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:24.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:24.060+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:24.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:24.060+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:24.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:24.061+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:24.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:24.083+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:24.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:24.083+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:24.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:24.083+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:24.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:24.084+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:25.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:25.029+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:25.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:25.029+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:25.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:25.029+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:25.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:25.030+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:25.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:24.889+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:25.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:24.889+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:25.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:24.889+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:25.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:24.890+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:25.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:25.076+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:25.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:25.076+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:25.285 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:25.076+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:25.285 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:25.076+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:25.736 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:15:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:15:25.611+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 24903 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:15:26.006 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:25.864+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:26.007 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:25.864+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:26.007 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:25.864+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:26.007 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:25.865+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:26.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:26.006+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:26.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:26.006+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:26.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:26.006+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:26.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:26.007+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:26.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:26.097+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:26.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:26.097+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:26.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:26.097+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:26.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:26.098+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:27.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:26.994+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:27.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:26.994+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:27.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:26.994+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:27.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:26.995+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:27.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:27.091+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:27.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:27.091+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:27.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:27.091+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:27.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:27.092+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:27.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:26.907+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:27.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:26.907+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:27.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:26.907+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:27.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:26.907+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:28.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:28.036+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:28.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:28.036+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:28.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:28.036+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:28.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:28.037+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:28.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:28.076+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:28.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:28.076+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:28.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:28.076+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:28.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:28.076+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:28.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:27.951+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:28.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:27.951+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:28.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:27.951+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:28.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:27.951+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:29.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:29.118+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:29.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:29.118+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:29.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:29.118+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:29.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:29.118+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:29.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:29.051+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:29.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:29.051+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:29.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:29.051+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:29.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:29.052+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:29.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:28.992+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:29.285 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:28.992+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:29.285 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:28.992+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:29.285 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:28.993+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:30.187 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:30.075+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:30.187 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:30.075+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:30.187 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:30.075+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:30.188 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:30.076+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:30.188 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:30.040+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:30.188 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:30.040+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:30.189 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:30.040+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:30.189 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:30.041+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:30.189 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:30.076+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:30.189 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:30.076+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:30.189 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:30.076+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:30.190 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:30.077+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:30.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:15:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:15:30.614+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 24991 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:15:31.281 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:31.037+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:31.281 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:31.037+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:31.281 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:31.037+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:31.281 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:31.038+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:31.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:31.039+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:31.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:31.039+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:31.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:31.039+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:31.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:31.040+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:31.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:31.026+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:31.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:31.026+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:31.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:31.026+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:31.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:31.027+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:32.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:32.067+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:32.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:32.067+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:32.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:32.067+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:32.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:32.068+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:32.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:32.028+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:32.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:32.028+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:32.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:32.028+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:32.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:32.028+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:32.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:32.034+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:32.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:32.034+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:32.285 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:32.034+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:32.285 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:32.034+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:33.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:32.988+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:33.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:32.988+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:33.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:32.988+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:33.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:32.989+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:33.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:32.997+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:33.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:32.997+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:33.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:32.997+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:33.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:32.998+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:33.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:33.103+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:33.285 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:33.103+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:33.285 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:33.103+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:33.285 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:33.104+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:34.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:33.987+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:34.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:33.987+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:34.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:33.987+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:34.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:33.988+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:34.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:34.064+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:34.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:34.064+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:34.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:34.064+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:34.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:34.064+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:34.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:34.020+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:34.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:34.020+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:34.285 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:34.020+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:34.285 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:34.020+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:35.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:34.997+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:35.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:34.997+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:35.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:34.997+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:35.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:34.997+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:35.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:34.977+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:35.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:34.977+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:35.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:34.977+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:35.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:34.977+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:35.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:35.023+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:35.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:35.023+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:35.285 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:35.023+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:35.285 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:35.024+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:36.019 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:15:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:15:35.617+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 25079 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:15:36.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:36.034+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:36.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:36.034+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:36.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:36.034+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:36.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:36.035+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:36.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:36.019+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:36.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:36.019+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:36.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:36.019+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:36.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:36.020+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:36.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:36.045+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:36.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:36.045+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:36.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:36.045+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:36.285 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:36.046+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:37.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:37.067+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:37.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:37.067+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:37.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:37.067+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:37.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:37.068+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:37.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:37.047+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:37.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:37.047+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:37.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:37.047+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:37.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:37.049+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:37.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:37.058+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:37.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:37.058+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:37.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:37.058+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:37.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:37.058+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:38.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:38.116+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:38.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:38.116+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:38.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:38.116+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:38.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:38.117+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:38.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:38.027+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:38.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:38.027+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:38.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:38.027+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:38.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:38.028+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:38.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:38.070+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:38.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:38.070+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:38.285 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:38.070+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:38.285 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:38.070+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:39.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:39.123+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:39.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:39.123+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:39.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:39.123+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:39.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:39.124+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:39.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:39.067+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:39.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:39.067+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:39.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:39.067+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:39.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:39.068+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:39.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:39.050+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:39.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:39.050+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:39.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:39.050+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:39.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:39.050+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:40.187 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:40.075+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:40.187 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:40.075+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:40.187 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:40.075+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:40.187 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:40.075+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:40.188 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:40.087+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:40.188 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:40.087+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:40.188 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:40.087+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:40.188 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:40.088+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:40.189 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:40.074+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:40.189 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:40.074+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:40.189 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:40.074+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:40.189 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:40.075+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:40.746 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:15:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:15:40.620+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 25167 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:15:41.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:41.049+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:41.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:41.049+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:41.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:41.049+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:41.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:41.050+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:41.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:41.097+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:41.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:41.097+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:41.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:41.097+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:41.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:41.098+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:41.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:41.035+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:41.535 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:41.035+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:41.535 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:41.035+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:41.535 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:41.036+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:42.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:42.061+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:42.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:42.061+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:42.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:42.061+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:42.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:42.062+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:42.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:42.060+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:42.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:42.060+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:42.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:42.060+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:42.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:42.061+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:42.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:42.073+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:42.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:42.073+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:42.535 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:42.073+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:42.535 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:42.073+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:43.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:43.101+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:43.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:43.101+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:43.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:43.101+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:43.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:43.102+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:43.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:43.067+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:43.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:43.067+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:43.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:43.067+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:43.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:43.067+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:43.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:43.119+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:43.535 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:43.119+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:43.535 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:43.119+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:43.535 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:43.119+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:44.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:44.104+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:44.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:44.104+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:44.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:44.104+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:44.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:44.104+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:44.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:44.091+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:44.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:44.091+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:44.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:44.091+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:44.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:44.091+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:44.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:44.117+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:44.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:44.117+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:44.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:44.117+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:44.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:44.117+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:45.433 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:45.133+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:45.434 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:45.133+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:45.434 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:45.133+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:45.434 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:45.134+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:45.434 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:45.079+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:45.435 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:45.079+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:45.435 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:45.079+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:45.435 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:45.080+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:45.435 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:45.109+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:45.435 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:45.109+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:45.435 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:45.109+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:45.436 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:45.109+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:45.746 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:15:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:15:45.623+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 25257 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:15:46.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:46.075+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:46.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:46.075+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:46.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:46.075+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:46.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:46.075+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:46.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:46.088+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:46.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:46.088+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:46.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:46.088+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:46.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:46.088+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:46.535 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:46.128+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:46.535 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:46.128+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:46.535 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:46.128+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:46.536 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:46.128+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:47.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:47.118+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:47.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:47.118+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:47.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:47.118+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:47.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:47.119+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:47.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:47.097+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:47.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:47.097+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:47.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:47.097+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:47.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:47.097+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:47.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:47.167+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:47.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:47.167+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:47.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:47.167+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:47.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:47.167+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:48.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:48.102+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:48.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:48.102+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:48.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:48.102+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:48.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:48.103+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:48.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:48.141+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:48.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:48.141+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:48.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:48.141+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:48.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:48.141+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:48.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:48.138+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:48.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:48.138+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:48.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:48.138+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:48.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:48.139+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:49.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:49.086+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:49.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:49.086+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:49.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:49.086+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:49.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:49.087+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:49.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:49.103+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:49.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:49.103+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:49.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:49.103+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:49.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:49.104+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:49.535 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:49.163+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:49.535 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:49.163+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:49.535 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:49.163+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:49.535 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:49.164+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:50.137 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:50.103+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:50.138 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:50.103+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:50.138 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:50.103+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:50.138 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:50.104+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:50.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:50.138+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:50.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:50.138+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:50.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:50.138+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:50.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:50.139+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:50.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:50.187+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:50.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:50.187+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:50.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:50.187+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:50.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:50.188+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:51.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:15:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:15:50.627+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 25346 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:15:51.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:51.142+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:51.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:51.142+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:51.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:51.142+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:51.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:51.142+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:51.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:51.149+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:51.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:51.149+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:51.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:51.149+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:51.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:51.150+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:51.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:51.182+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:51.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:51.182+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:51.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:51.182+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:51.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:51.183+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:52.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:52.176+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:52.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:52.176+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:52.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:52.176+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:52.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:52.176+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:52.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:52.115+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:52.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:52.115+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:52.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:52.115+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:52.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:52.115+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:52.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:52.220+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:52.535 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:52.220+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:52.535 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:52.220+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:52.535 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:52.221+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:53.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:53.256+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:53.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:53.256+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:53.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:53.256+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:53.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:53.257+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:53.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:53.097+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:53.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:53.097+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:53.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:53.097+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:53.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:53.098+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:53.535 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:53.169+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:53.535 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:53.169+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:53.535 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:53.169+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:53.535 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:53.169+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:54.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:54.287+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:54.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:54.287+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:54.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:54.287+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:54.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:54.288+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:54.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:54.124+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:54.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:54.124+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:54.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:54.124+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:54.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:54.125+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:54.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:54.117+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:54.535 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:54.117+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:54.535 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:54.117+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:54.535 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:54.118+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:55.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:55.082+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:55.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:55.082+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:55.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:55.082+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:55.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:55.083+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:55.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:55.314+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:55.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:55.314+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:55.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:55.314+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:55.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:55.315+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:55.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:55.152+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:55.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:55.152+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:55.535 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:55.152+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:55.535 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:55.153+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:56.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:15:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:15:55.629+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 25435 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:15:56.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:56.122+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:56.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:56.122+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:56.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:56.122+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:56.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:56.123+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:56.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:56.303+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:56.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:56.303+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:56.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:56.303+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:56.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:56.304+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:56.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:56.120+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:56.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:56.120+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:56.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:56.120+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:56.535 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:56.120+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:57.352 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:57.079+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:57.352 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:57.079+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:57.352 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:57.079+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:57.352 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:57.080+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:57.353 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:57.096+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:57.353 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:57.096+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:57.353 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:57.096+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:57.353 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:57.097+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:57.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:57.352+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:57.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:57.352+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:57.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:57.352+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:57.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:57.353+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:58.373 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:58.117+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:58.373 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:58.117+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:58.373 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:58.117+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:58.373 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:58.118+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:58.373 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:58.131+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:58.374 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:58.131+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:58.374 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:58.131+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:58.374 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:58.132+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:58.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:58.373+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:58.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:58.373+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:58.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:58.373+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:58.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:58.374+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:59.334 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:59.174+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:59.334 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:59.174+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:59.334 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:59.174+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:15:59.334 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:15:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:15:59.174+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:59.335 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:59.082+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:59.335 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:59.082+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:59.335 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:59.082+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:15:59.335 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:15:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:15:59.082+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:15:59.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:59.334+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:59.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:59.334+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:59.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:59.334+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:15:59.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:15:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:15:59.334+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:00.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:00.198+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:00.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:00.198+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:00.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:00.198+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:00.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:00.199+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:00.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:00.294+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:00.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:00.294+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:00.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:00.294+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:00.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:00.295+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:00.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:00.111+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:00.535 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:00.111+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:00.535 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:00.111+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:00.535 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:00.113+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:01.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:16:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:16:00.632+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 25524 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:16:01.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:01.337+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:01.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:01.337+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:01.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:01.337+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:01.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:01.338+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:01.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:01.234+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:01.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:01.234+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:01.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:01.234+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:01.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:01.235+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:01.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:01.094+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:01.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:01.094+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:01.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:01.094+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:01.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:01.095+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:02.255 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:02.095+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:02.255 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:02.095+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:02.256 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:02.095+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:02.256 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:02.096+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:02.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:02.255+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:02.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:02.255+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:02.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:02.255+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:02.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:02.256+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:02.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:02.294+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:02.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:02.294+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:02.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:02.295+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:02.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:02.295+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:03.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:03.093+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:03.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:03.093+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:03.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:03.093+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:03.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:03.094+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:03.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:03.295+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:03.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:03.295+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:03.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:03.295+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:03.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:03.295+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:03.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:03.311+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:03.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:03.311+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:03.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:03.311+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:03.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:03.312+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:04.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:04.311+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:04.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:04.311+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:04.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:04.311+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:04.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:04.311+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:04.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:04.294+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:04.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:04.294+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:04.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:04.294+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:04.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:04.295+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:04.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:04.132+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:04.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:04.132+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:04.535 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:04.132+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:04.535 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:04.133+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:05.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:05.127+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:05.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:05.127+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:05.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:05.127+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:05.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:05.128+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:05.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:05.304+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:05.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:05.304+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:05.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:05.304+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:05.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:05.305+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:05.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:05.262+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:05.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:05.262+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:05.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:05.262+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:05.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:05.263+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:06.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:16:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:16:05.634+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 25614 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:16:06.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:06.290+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:06.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:06.290+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:06.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:06.290+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:06.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:06.291+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:06.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:06.282+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:06.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:06.282+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:06.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:06.282+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:06.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:06.283+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:06.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:06.106+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:06.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:06.106+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:06.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:06.106+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:06.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:06.108+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:07.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:07.294+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:07.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:07.294+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:07.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:07.294+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:07.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:07.294+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:07.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:07.321+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:07.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:07.321+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:07.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:07.321+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:07.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:07.322+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:07.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:07.099+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:07.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:07.099+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:07.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:07.099+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:07.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:07.100+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:08.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:08.308+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:08.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:08.308+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:08.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:08.308+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:08.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:08.308+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:08.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:08.129+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:08.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:08.129+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:08.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:08.129+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:08.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:08.130+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:08.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:08.319+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:08.535 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:08.319+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:08.535 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:08.319+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:08.535 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:08.320+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:09.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:09.300+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:09.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:09.300+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:09.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:09.300+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:09.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:09.300+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:09.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:09.301+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:09.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:09.301+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:09.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:09.301+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:09.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:09.302+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:09.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:09.156+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:09.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:09.156+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:09.535 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:09.156+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:09.535 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:09.157+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:10.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:10.341+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:10.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:10.341+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:10.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:10.341+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:10.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:10.342+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:10.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:10.118+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:10.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:10.118+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:10.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:10.118+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:10.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:10.119+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:10.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:10.296+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:10.535 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:10.296+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:10.535 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:10.296+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:10.535 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:10.296+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:11.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:16:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:16:10.637+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 25703 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:16:11.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:11.349+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:11.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:11.349+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:11.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:11.349+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:11.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:11.350+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:11.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:11.321+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:11.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:11.321+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:11.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:11.321+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:11.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:11.322+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:11.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:11.133+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:11.535 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:11.133+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:11.535 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:11.133+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:11.535 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:11.134+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:12.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:12.363+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:12.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:12.363+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:12.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:12.363+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:12.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:12.363+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:12.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:12.311+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:12.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:12.311+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:12.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:12.311+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:12.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:12.312+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:12.535 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:12.151+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:12.535 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:12.151+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:12.535 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:12.151+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:12.535 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:12.152+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:13.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:13.327+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:13.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:13.327+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:13.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:13.327+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:13.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:13.327+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:13.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:13.313+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:13.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:13.313+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:13.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:13.313+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:13.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:13.314+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:13.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:13.149+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:13.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:13.149+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:13.535 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:13.149+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:13.535 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:13.150+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:14.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:14.297+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:14.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:14.297+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:14.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:14.297+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:14.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:14.298+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:14.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:14.121+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:14.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:14.121+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:14.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:14.121+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:14.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:14.122+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:14.535 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:14.310+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:14.535 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:14.310+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:14.535 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:14.310+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:14.535 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:14.310+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:15.350 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:15.324+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:15.350 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:15.324+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:15.350 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:15.324+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:15.350 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:15.325+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:15.350 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:15.091+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:15.351 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:15.091+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:15.351 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:15.091+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:15.351 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:15.092+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:15.640 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:15.349+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:15.641 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:15.349+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:15.641 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:15.349+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:15.641 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:15.350+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:16.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:16:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:16:15.640+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 25792 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:16:16.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:16.370+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:16.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:16.370+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:16.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:16.370+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:16.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:16.371+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:16.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:16.301+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:16.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:16.301+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:16.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:16.301+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:16.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:16.302+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:16.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:16.127+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:16.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:16.127+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:16.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:16.127+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:16.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:16.128+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:17.382 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:17.335+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:17.382 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:17.335+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:17.382 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:17.335+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:17.382 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:17.336+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:17.382 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:17.129+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:17.383 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:17.129+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:17.383 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:17.129+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:17.383 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:17.130+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:17.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:17.381+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:17.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:17.381+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:17.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:17.381+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:17.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:17.382+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:18.363 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:18.291+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:18.363 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:18.292+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:18.363 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:18.292+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:18.363 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:18.292+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:18.364 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:18.103+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:18.364 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:18.103+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:18.364 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:18.103+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:18.364 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:18.104+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:18.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:18.362+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:18.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:18.362+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:18.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:18.362+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:18.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:18.363+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:19.394 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:19.342+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:19.395 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:19.342+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:19.395 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:19.342+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:19.395 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:19.343+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:19.395 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:19.120+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:19.395 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:19.120+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:19.396 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:19.120+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:19.396 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:19.120+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:19.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:19.394+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:19.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:19.394+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:19.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:19.394+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:19.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:19.395+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:20.432 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:20.363+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:20.433 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:20.363+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:20.433 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:20.363+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:20.433 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:20.364+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:20.434 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:20.351+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:20.434 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:20.351+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:20.434 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:20.351+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:20.434 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:20.352+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:20.435 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:20.121+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:20.435 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:20.121+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:20.435 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:20.121+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:20.435 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:20.122+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:20.764 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:16:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:16:20.643+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 25881 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:16:21.360 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:21.339+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:21.360 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:21.339+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:21.361 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:21.339+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:21.361 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:21.340+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:21.361 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:21.096+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:21.361 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:21.096+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:21.361 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:21.096+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:21.362 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:21.097+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:21.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:21.360+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:21.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:21.360+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:21.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:21.360+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:21.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:21.361+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:22.394 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:22.086+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:22.394 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:22.086+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:22.394 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:22.086+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:22.395 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:22.087+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:22.397 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:22.314+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:22.398 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:22.314+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:22.398 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:22.314+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:22.398 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:22.315+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:22.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:22.393+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:22.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:22.393+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:22.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:22.393+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:22.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:22.394+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:23.348 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:23.058+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:23.349 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:23.058+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:23.349 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:23.058+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:23.349 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:23.059+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:23.618 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:23.348+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:23.619 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:23.348+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:23.619 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:23.348+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:23.619 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:23.349+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:23.619 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:23.357+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:23.619 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:23.357+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:23.619 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:23.357+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:23.620 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:23.358+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:24.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:24.315+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:24.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:24.315+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:24.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:24.315+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:24.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:24.315+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:24.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:24.319+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:24.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:24.319+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:24.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:24.319+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:24.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:24.320+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:24.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:24.084+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:24.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:24.084+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:24.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:24.084+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:24.535 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:24.085+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:25.323 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:25.068+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:25.323 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:25.068+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:25.323 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:25.068+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:25.323 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:25.069+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:25.647 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:25.369+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:25.647 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:25.369+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:25.647 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:25.369+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:25.648 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:25.370+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:25.648 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:25.322+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:25.648 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:25.322+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:25.648 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:25.322+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:25.649 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:25.323+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:25.962 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:16:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:16:25.646+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 25971 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:16:26.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:26.037+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:26.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:26.037+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:26.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:26.037+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:26.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:26.038+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:26.576 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:26.306+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:26.576 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:26.306+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:26.577 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:26.306+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:26.577 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:26.307+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:26.577 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:26.379+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:26.577 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:26.379+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:26.577 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:26.379+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:26.577 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:26.379+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:27.342 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:27.290+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:27.342 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:27.290+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:27.343 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:27.290+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:27.343 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:27.290+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:27.343 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:27.070+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:27.343 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:27.070+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:27.343 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:27.070+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:27.343 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:27.071+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:27.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:27.341+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:27.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:27.341+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:27.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:27.341+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:27.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:27.342+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:28.245+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:28.245+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:28.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:28.245+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:28.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:28.246+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:28.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:28.070+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:28.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:28.070+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:28.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:28.070+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:28.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:28.070+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:28.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:28.311+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:28.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:28.311+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:28.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:28.311+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:28.535 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:28.312+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:29.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:29.261+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:29.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:29.261+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:29.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:29.261+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:29.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:29.262+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:29.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:29.277+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:29.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:29.277+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:29.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:29.277+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:29.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:29.277+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:29.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:29.049+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:29.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:29.049+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:29.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:29.049+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:29.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:29.050+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:30.191 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:30.074+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:30.191 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:30.074+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:30.191 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:30.074+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:30.192 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:30.074+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:30.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:30.297+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:30.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:30.297+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:30.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:30.297+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:30.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:30.297+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:30.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:30.239+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:30.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:30.239+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:30.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:30.239+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:30.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:30.240+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:31.002 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:16:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:16:30.649+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 26060 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:16:31.268 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:31.061+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:31.268 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:31.061+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:31.268 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:31.061+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:31.268 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:31.063+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:31.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:31.267+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:31.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:31.267+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:31.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:31.267+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:31.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:31.268+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:31.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:31.285+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:31.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:31.285+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:31.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:31.285+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:31.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:31.286+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:32.331 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:32.276+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:32.332 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:32.276+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:32.332 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:32.276+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:32.332 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:32.276+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:32.332 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:32.050+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:32.333 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:32.050+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:32.333 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:32.050+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:32.333 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:32.051+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:32.720 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:32.331+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:32.721 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:32.331+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:32.721 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:32.331+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:32.721 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:32.331+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:33.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:33.278+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:33.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:33.278+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:33.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:33.278+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:33.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:33.279+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:33.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:33.312+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:33.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:33.312+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:33.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:33.312+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:33.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:33.313+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:33.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:33.076+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:33.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:33.076+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:33.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:33.076+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:33.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:33.077+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:34.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:34.112+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:34.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:34.112+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:34.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:34.112+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:34.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:34.113+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:34.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:34.282+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:34.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:34.282+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:34.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:34.282+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:34.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:34.282+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:34.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:34.320+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:34.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:34.320+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:34.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:34.320+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:34.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:34.321+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:35.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:35.123+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:35.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:35.123+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:35.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:35.123+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:35.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:35.124+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:35.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:35.246+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:35.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:35.246+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:35.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:35.246+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:35.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:35.247+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:35.535 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:35.360+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:35.535 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:35.360+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:35.535 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:35.360+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:35.535 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:35.361+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:36.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:16:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:16:35.652+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 26149 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:16:36.329 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:36.281+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:36.329 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:36.281+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:36.330 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:36.281+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:36.330 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:36.282+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:36.330 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:36.102+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:36.330 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:36.102+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:36.331 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:36.102+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:36.331 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:36.103+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:36.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:36.328+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:36.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:36.328+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:36.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:36.328+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:36.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:36.329+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:37.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:37.322+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:37.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:37.322+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:37.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:37.322+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:37.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:37.323+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:37.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:37.354+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:37.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:37.354+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:37.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:37.354+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:37.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:37.355+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:37.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:37.140+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:37.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:37.140+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:37.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:37.140+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:37.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:37.141+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:38.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:38.188+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:38.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:38.188+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:38.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:38.188+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:38.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:38.189+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:38.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:38.289+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:38.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:38.289+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:38.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:38.289+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:38.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:38.290+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:38.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:38.396+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:38.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:38.396+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:38.535 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:38.396+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:38.535 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:38.397+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:39.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:39.192+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:39.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:39.192+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:39.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:39.192+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:39.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:39.193+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:39.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:39.282+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:39.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:39.282+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:39.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:39.282+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:39.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:39.283+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:39.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:39.429+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:39.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:39.429+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:39.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:39.429+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:39.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:39.430+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:40.454 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:40.222+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:40.454 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:40.222+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:40.455 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:40.222+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:40.455 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:40.223+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:40.455 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:40.242+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:40.455 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:40.242+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:40.455 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:40.242+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:40.456 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:40.243+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:40.781 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:16:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:16:40.655+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 26238 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:16:40.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:40.453+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:40.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:40.453+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:40.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:40.453+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:40.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:40.454+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:41.451 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:41.257+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:41.451 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:41.257+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:41.452 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:41.257+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:41.452 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:41.258+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:41.452 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:41.227+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:41.452 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:41.227+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:41.452 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:41.227+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:41.452 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:41.228+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:41.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:41.451+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:41.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:41.451+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:41.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:41.451+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:41.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:41.452+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:42.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:42.226+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:42.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:42.226+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:42.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:42.226+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:42.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:42.227+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:42.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:42.255+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:42.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:42.255+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:42.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:42.255+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:42.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:42.256+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:42.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:42.409+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:42.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:42.410+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:42.535 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:42.410+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:42.535 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:42.410+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:43.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:43.207+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:43.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:43.207+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:43.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:43.207+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:43.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:43.208+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:43.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:43.417+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:43.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:43.417+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:43.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:43.417+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:43.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:43.418+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:43.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:43.259+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:43.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:43.259+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:43.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:43.259+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:43.535 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:43.260+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:44.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:44.275+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:44.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:44.275+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:44.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:44.275+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:44.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:44.276+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:44.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:44.393+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:44.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:44.393+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:44.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:44.393+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:44.534 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:44.393+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:44.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:44.225+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:44.535 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:44.225+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:44.535 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:44.225+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:44.535 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:44.226+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:45.269 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:45.198+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:45.269 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:45.198+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:45.270 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:45.198+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:45.270 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:45.199+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:45.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:45.406+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:45.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:45.406+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:45.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:45.406+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:45.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:45.406+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:45.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:45.269+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:45.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:45.269+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:45.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:45.269+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:45.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:45.270+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:46.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:16:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:16:45.658+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 26328 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:16:46.437 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:46.169+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:46.437 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:46.169+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:46.438 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:46.169+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:46.438 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:46.170+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:46.438 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:46.305+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:46.438 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:46.305+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:46.438 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:46.305+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:46.439 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:46.306+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:46.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:46.436+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:46.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:46.436+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:46.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:46.436+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:46.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:46.437+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:47.467 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:47.153+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:47.467 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:47.153+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:47.467 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:47.153+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:47.467 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:47.154+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:47.467 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:47.301+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:47.467 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:47.301+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:47.468 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:47.301+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:47.468 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:47.302+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:47.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:47.466+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:47.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:47.466+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:47.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:47.466+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:47.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:47.467+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:48.469 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:48.277+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:48.469 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:48.277+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:48.470 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:48.277+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:48.470 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:48.278+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:48.470 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:48.138+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:48.470 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:48.138+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:48.470 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:48.138+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:48.470 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:48.139+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:48.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:48.468+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:48.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:48.468+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:48.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:48.468+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:48.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:48.469+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:49.489 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:49.142+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:49.490 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:49.142+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:49.490 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:49.142+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:49.490 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:49.143+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:49.491 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:49.230+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:49.491 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:49.230+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:49.491 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:49.230+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:49.491 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:49.231+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:49.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:49.489+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:49.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:49.489+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:49.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:49.489+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:49.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:49.489+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:50.529 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:50.271+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:50.529 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:50.271+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:50.529 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:50.271+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:50.530 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:50.272+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:50.530 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:50.139+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:50.530 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:50.139+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:50.530 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:50.139+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:50.531 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:50.140+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:50.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:16:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:16:50.661+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 26417 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:16:50.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:50.528+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:50.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:50.528+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:50.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:50.528+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:50.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:50.529+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:51.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:51.291+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:51.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:51.291+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:51.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:51.291+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:51.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:51.291+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:51.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:51.166+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:51.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:51.166+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:51.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:51.166+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:51.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:51.166+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:51.552 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:51.550+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:51.553 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:51.550+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:51.553 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:51.550+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:51.553 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:51.552+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:52.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:52.247+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:52.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:52.247+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:52.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:52.247+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:52.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:52.248+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:52.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:52.137+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:52.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:52.137+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:52.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:52.137+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:52.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:52.138+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:53.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:52.590+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:53.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:52.590+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:53.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:52.590+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:53.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:52.591+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:53.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:53.227+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:53.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:53.227+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:53.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:53.227+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:53.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:53.228+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:53.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:53.157+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:53.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:53.157+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:53.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:53.157+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:53.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:53.158+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:54.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:53.580+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:54.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:53.580+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:54.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:53.580+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:54.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:53.581+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:54.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:54.272+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:54.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:54.272+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:54.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:54.272+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:54.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:54.273+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:54.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:54.146+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:54.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:54.146+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:54.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:54.146+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:54.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:54.147+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:55.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:54.574+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:55.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:54.574+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:55.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:54.574+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:55.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:54.575+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:55.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:55.229+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:55.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:55.229+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:55.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:55.229+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:55.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:55.230+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:55.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:55.104+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:55.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:55.104+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:55.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:55.104+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:55.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:55.105+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:56.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:16:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:16:55.664+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 26506 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:16:56.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:55.543+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:56.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:55.543+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:56.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:55.543+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:56.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:55.544+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:56.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:56.138+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:56.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:56.138+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:56.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:56.138+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:56.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:56.139+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:56.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:56.251+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:56.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:56.251+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:56.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:56.251+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:56.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:56.251+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:57.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:56.545+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:57.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:56.545+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:57.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:56.545+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:57.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:56.545+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:57.514 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:57.135+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:57.514 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:57.135+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:57.515 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:57.135+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:57.515 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:57.136+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:57.515 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:57.275+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:57.515 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:57.275+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:57.516 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:57.275+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:57.516 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:57.276+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:57.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:57.513+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:57.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:57.513+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:57.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:57.513+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:57.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:57.514+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:58.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:58.143+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:58.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:58.143+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:58.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:58.143+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:58.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:58.144+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:58.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:58.249+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:58.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:58.249+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:58.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:58.249+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:58.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:58.250+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:59.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:58.555+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:59.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:58.555+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:59.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:58.555+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:16:59.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:58.556+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:59.535 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:59.124+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:59.535 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:59.124+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:59.535 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:59.124+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:16:59.535 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:16:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:16:59.124+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:16:59.536 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:59.299+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:59.536 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:59.299+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:59.536 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:59.299+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:16:59.536 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:16:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:16:59.300+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:00.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:59.604+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:00.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:59.604+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:00.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:59.604+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:00.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:16:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:16:59.605+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:00.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:00.271+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:00.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:00.271+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:00.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:00.271+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:00.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:00.272+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:00.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:00.155+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:00.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:00.155+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:00.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:00.155+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:00.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:00.156+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:01.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:17:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:17:00.667+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 26595 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:17:01.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:00.567+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:01.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:00.567+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:01.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:00.567+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:01.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:00.568+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:01.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:01.162+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:01.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:01.162+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:01.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:01.163+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:01.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:01.163+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:01.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:01.288+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:01.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:01.288+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:01.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:01.288+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:01.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:01.289+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:02.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:01.600+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:02.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:01.600+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:02.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:01.600+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:02.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:01.600+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:02.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:02.126+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:02.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:02.126+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:02.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:02.126+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:02.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:02.126+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:02.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:02.250+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:02.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:02.250+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:02.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:02.250+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:02.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:02.250+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:03.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:02.620+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:03.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:02.620+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:03.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:02.620+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:03.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:02.621+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:03.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:03.111+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:03.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:03.111+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:03.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:03.111+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:03.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:03.112+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:03.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:03.219+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:03.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:03.219+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:03.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:03.219+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:03.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:03.220+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:04.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:03.643+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:04.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:03.643+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:04.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:03.643+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:04.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:03.643+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:04.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:04.109+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:04.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:04.109+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:04.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:04.109+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:04.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:04.110+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:04.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:04.248+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:04.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:04.248+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:04.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:04.248+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:04.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:04.249+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:05.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:04.683+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:05.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:04.683+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:05.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:04.683+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:05.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:04.683+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:05.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:05.157+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:05.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:05.157+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:05.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:05.157+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:05.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:05.158+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:05.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:05.232+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:05.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:05.232+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:05.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:05.232+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:05.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:05.233+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:06.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:17:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:17:05.670+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 26685 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:17:06.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:05.670+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:06.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:05.670+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:06.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:05.670+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:06.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:05.671+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:06.513 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:06.194+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:06.513 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:06.194+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:06.513 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:06.194+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:06.513 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:06.195+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:06.514 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:06.266+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:06.514 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:06.266+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:06.514 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:06.266+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:06.515 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:06.266+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:06.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:06.710+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:06.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:06.710+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:06.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:06.710+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:06.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:06.710+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:07.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:07.167+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:07.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:07.167+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:07.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:07.167+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:07.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:07.168+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:07.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:07.217+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:07.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:07.217+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:07.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:07.217+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:07.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:07.217+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:08.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:07.755+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:08.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:07.755+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:08.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:07.755+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:08.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:07.757+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:08.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:08.185+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:08.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:08.185+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:08.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:08.185+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:08.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:08.185+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:08.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:08.182+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:08.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:08.182+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:08.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:08.182+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:08.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:08.182+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:09.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:08.726+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:09.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:08.726+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:09.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:08.726+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:09.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:08.727+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:09.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:09.209+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:09.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:09.209+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:09.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:09.209+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:09.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:09.210+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:09.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:09.147+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:09.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:09.147+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:09.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:09.147+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:09.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:09.148+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:10.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:09.753+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:10.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:09.753+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:10.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:09.753+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:10.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:09.754+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:10.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:10.196+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:10.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:10.196+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:10.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:10.196+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:10.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:10.197+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:10.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:10.184+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:10.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:10.184+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:10.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:10.184+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:10.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:10.185+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:11.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:17:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:17:10.674+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 26774 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:17:11.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:10.769+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:11.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:10.769+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:11.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:10.769+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:11.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:10.769+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:11.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:11.163+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:11.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:11.163+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:11.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:11.163+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:11.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:11.163+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:11.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:11.159+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:11.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:11.159+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:11.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:11.159+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:11.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:11.160+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:12.110 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:11.798+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:12.110 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:11.798+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:12.111 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:11.798+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:12.111 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:11.799+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:12.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:12.146+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:12.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:12.146+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:12.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:12.146+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:12.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:12.147+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:12.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:12.111+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:12.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:12.111+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:12.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:12.111+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:12.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:12.111+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:13.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:12.805+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:13.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:12.805+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:13.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:12.805+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:13.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:12.806+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:13.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:13.162+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:13.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:13.162+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:13.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:13.162+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:13.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:13.163+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:13.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:13.127+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:13.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:13.127+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:13.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:13.127+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:13.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:13.128+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:14.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:13.792+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:14.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:13.792+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:14.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:13.792+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:14.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:13.792+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:14.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:14.199+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:14.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:14.199+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:14.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:14.199+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:14.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:14.200+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:14.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:14.169+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:14.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:14.169+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:14.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:14.169+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:14.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:14.169+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:15.155 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:14.806+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:15.155 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:14.806+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:15.155 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:14.806+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:15.155 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:14.807+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:15.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:15.207+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:15.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:15.207+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:15.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:15.207+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:15.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:15.208+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:15.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:15.155+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:15.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:15.155+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:15.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:15.155+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:15.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:15.156+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:16.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:17:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:17:15.677+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 26863 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:17:16.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:15.772+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:16.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:15.772+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:16.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:15.772+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:16.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:15.773+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:16.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:16.179+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:16.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:16.179+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:16.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:16.179+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:16.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:16.180+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:16.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:16.214+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:16.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:16.214+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:16.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:16.214+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:16.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:16.214+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:17.223 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:16.813+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:17.223 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:16.813+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:17.223 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:16.813+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:17.223 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:16.814+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:17.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:17.223+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:17.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:17.223+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:17.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:17.223+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:17.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:17.224+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:17.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:17.257+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:17.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:17.257+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:17.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:17.257+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:17.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:17.258+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:18.233 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:17.785+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:18.233 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:17.785+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:18.233 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:17.785+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:18.233 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:17.786+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:18.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:18.239+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:18.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:18.239+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:18.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:18.239+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:18.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:18.239+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:18.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:18.233+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:18.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:18.233+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:18.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:18.233+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:18.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:18.234+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:19.213 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:18.805+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:19.213 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:18.805+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:19.213 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:18.805+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:19.213 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:18.807+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:19.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:19.229+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:19.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:19.229+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:19.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:19.229+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:19.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:19.230+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:19.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:19.213+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:19.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:19.213+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:19.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:19.213+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:19.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:19.214+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:20.191 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:19.815+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:20.192 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:19.815+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:20.192 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:19.815+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:20.192 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:19.815+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:20.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:20.226+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:20.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:20.226+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:20.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:20.226+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:20.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:20.227+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:20.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:20.192+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:20.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:20.192+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:20.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:20.192+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:20.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:20.192+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:21.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:17:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:17:20.680+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 26952 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:17:21.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:20.781+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:21.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:20.781+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:21.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:20.781+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:21.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:20.782+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:21.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:21.187+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:21.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:21.187+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:21.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:21.187+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:21.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:21.187+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:21.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:21.213+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:21.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:21.213+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:21.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:21.213+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:21.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:21.213+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:22.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:21.776+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:22.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:21.776+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:22.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:21.776+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:22.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:21.777+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:22.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:22.222+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:22.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:22.222+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:22.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:22.222+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:22.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:22.223+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:22.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:22.226+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:22.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:22.226+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:22.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:22.226+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:22.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:22.227+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:23.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:22.751+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:23.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:22.751+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:23.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:22.751+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:23.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:22.752+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:23.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:23.217+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:23.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:23.217+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:23.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:23.217+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:23.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:23.218+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:23.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:23.235+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:23.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:23.235+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:23.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:23.235+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:23.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:23.236+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:24.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:23.749+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:24.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:23.749+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:24.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:23.749+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:24.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:23.750+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:24.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:24.231+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:24.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:24.231+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:24.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:24.231+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:24.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:24.231+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:24.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:24.231+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:24.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:24.231+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:24.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:24.231+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:24.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:24.232+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:25.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:24.722+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:25.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:24.722+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:25.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:24.722+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:25.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:24.723+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:25.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:25.232+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:25.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:25.232+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:25.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:25.232+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:25.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:25.232+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:25.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:25.230+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:25.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:25.230+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:25.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:25.230+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:25.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:25.231+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:26.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:17:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:17:25.682+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 27041 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:17:26.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:25.689+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:26.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:25.689+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:26.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:25.689+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:26.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:25.690+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:26.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:26.238+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:26.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:26.238+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:26.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:26.238+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:26.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:26.239+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:26.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:26.201+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:26.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:26.201+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:26.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:26.201+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:26.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:26.203+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:27.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:26.738+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:27.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:26.738+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:27.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:26.738+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:27.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:26.739+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:27.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:27.274+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:27.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:27.274+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:27.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:27.274+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:27.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:27.275+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:27.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:27.168+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:27.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:27.168+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:27.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:27.168+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:27.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:27.169+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:28.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:27.692+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:28.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:27.692+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:28.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:27.692+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:28.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:27.693+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:28.306+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:28.306+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:28.306+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:28.307+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:28.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:28.136+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:28.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:28.136+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:28.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:28.136+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:28.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:28.136+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:29.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:28.719+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:29.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:28.719+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:29.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:28.719+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:29.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:28.719+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:29.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:29.322+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:29.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:29.322+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:29.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:29.322+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:29.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:29.322+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:29.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:29.142+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:29.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:29.142+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:29.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:29.142+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:29.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:29.143+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:30.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:29.682+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:30.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:29.682+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:30.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:29.682+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:30.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:29.683+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:30.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:30.320+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:30.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:30.320+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:30.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:30.320+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:30.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:30.321+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:30.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:30.146+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:30.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:30.146+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:30.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:30.146+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:30.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:30.147+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:31.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:17:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:17:30.685+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 27129 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:17:31.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:30.639+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:31.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:30.639+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:31.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:30.639+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:31.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:30.639+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:31.491 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:31.334+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:31.491 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:31.334+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:31.491 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:31.334+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:31.491 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:31.335+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:31.492 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:31.177+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:31.492 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:31.177+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:31.492 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:31.177+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:31.492 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:31.177+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:31.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:31.674+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:31.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:31.674+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:31.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:31.674+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:31.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:31.674+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:32.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:32.381+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:32.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:32.381+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:32.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:32.381+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:32.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:32.382+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:32.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:32.227+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:32.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:32.227+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:32.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:32.227+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:32.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:32.228+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:33.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:32.685+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:33.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:32.685+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:33.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:32.685+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:33.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:32.686+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:33.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:33.358+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:33.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:33.358+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:33.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:33.358+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:33.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:33.359+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:33.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:33.180+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:33.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:33.180+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:33.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:33.180+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:33.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:33.180+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:34.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:33.716+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:34.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:33.717+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:34.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:33.717+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:34.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:33.717+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:34.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:34.349+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:34.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:34.349+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:34.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:34.349+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:34.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:34.350+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:34.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:34.167+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:34.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:34.167+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:34.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:34.167+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:34.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:34.167+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:35.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:34.722+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:35.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:34.722+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:35.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:34.722+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:35.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:34.723+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:35.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:35.353+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:35.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:35.353+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:35.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:35.353+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:35.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:35.354+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:35.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:35.147+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:35.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:35.147+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:35.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:35.147+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:35.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:35.149+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:36.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:17:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:17:35.688+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 27218 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:17:36.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:35.709+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:36.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:35.709+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:36.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:35.709+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:36.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:35.710+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:36.531 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:36.346+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:36.531 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:36.346+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:36.531 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:36.347+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:36.531 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:36.347+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:36.531 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:36.177+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:36.531 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:36.177+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:36.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:36.177+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:36.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:36.178+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:36.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:36.664+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:36.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:36.664+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:36.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:36.664+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:36.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:36.665+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:37.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:37.352+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:37.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:37.352+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:37.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:37.352+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:37.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:37.353+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:37.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:37.181+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:37.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:37.181+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:37.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:37.181+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:37.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:37.182+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:38.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:37.657+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:38.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:37.657+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:38.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:37.657+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:38.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:37.658+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:38.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:38.177+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:38.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:38.177+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:38.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:38.177+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:38.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:38.178+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:38.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:38.395+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:38.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:38.395+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:38.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:38.395+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:38.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:38.396+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:38.646+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:38.646+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:38.646+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:39.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:38.647+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:39.416 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:39.154+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:39.416 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:39.154+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:39.417 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:39.154+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:39.417 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:39.155+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:39.666 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:39.665+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:39.666 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:39.665+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:39.667 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:39.415+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:39.667 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:39.415+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:39.667 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:39.415+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:39.667 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:39.416+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:40.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:39.665+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:40.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:39.666+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:40.376 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:40.106+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:40.376 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:40.106+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:40.376 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:40.106+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:40.377 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:40.106+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:40.670 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:40.375+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:40.670 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:40.375+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:40.670 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:40.375+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:40.671 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:40.376+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:41.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:17:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:17:40.691+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 27307 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:17:41.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:40.669+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:41.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:40.669+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:41.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:40.669+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:41.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:40.670+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:41.397 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:41.101+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:41.398 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:41.101+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:41.398 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:41.101+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:41.398 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:41.102+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:41.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:41.397+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:41.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:41.397+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:41.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:41.397+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:41.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:41.398+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:41.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:41.645+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:41.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:41.645+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:41.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:41.645+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:41.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:41.646+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:42.378 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:42.071+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:42.378 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:42.071+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:42.378 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:42.071+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:42.378 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:42.072+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:42.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:42.377+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:42.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:42.377+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:42.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:42.377+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:42.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:42.378+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:42.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:42.606+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:42.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:42.606+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:42.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:42.606+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:42.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:42.606+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:43.380 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:43.063+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:43.380 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:43.063+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:43.380 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:43.063+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:43.381 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:43.064+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:43.646 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:43.380+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:43.646 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:43.380+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:43.646 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:43.380+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:43.646 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:43.381+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:44.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:43.645+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:44.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:43.645+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:44.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:43.645+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:44.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:43.646+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:44.379 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:44.068+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:44.379 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:44.068+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:44.379 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:44.068+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:44.380 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:44.068+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:44.683 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:44.378+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:44.683 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:44.378+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:44.683 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:44.379+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:44.683 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:44.379+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:45.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:44.682+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:45.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:44.682+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:45.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:44.682+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:45.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:44.683+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:45.358 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:45.111+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:45.359 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:45.111+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:45.359 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:45.111+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:45.359 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:45.112+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:45.694 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:45.358+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:45.695 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:45.358+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:45.695 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:45.358+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:45.695 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:45.359+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:46.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:17:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:17:45.694+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 27397 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:17:46.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:45.706+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:46.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:45.706+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:46.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:45.706+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:46.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:45.707+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:46.373 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:46.091+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:46.373 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:46.091+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:46.373 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:46.091+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:46.373 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:46.091+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:46.681 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:46.372+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:46.681 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:46.372+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:46.681 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:46.372+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:46.682 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:46.373+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:47.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:46.680+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:47.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:46.680+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:47.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:46.680+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:47.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:46.681+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:47.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:47.112+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:47.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:47.112+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:47.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:47.112+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:47.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:47.113+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:47.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:47.334+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:47.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:47.334+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:47.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:47.334+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:47.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:47.335+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:48.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:47.681+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:48.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:47.681+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:48.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:47.681+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:48.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:47.683+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:48.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:48.101+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:48.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:48.101+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:48.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:48.101+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:48.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:48.102+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:48.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:48.332+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:48.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:48.332+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:48.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:48.332+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:48.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:48.333+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:49.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:48.724+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:49.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:48.724+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:49.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:48.724+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:49.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:48.725+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:49.323 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:49.056+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:49.324 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:49.056+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:49.324 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:49.056+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:49.324 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:49.056+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:49.691 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:49.323+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:49.691 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:49.323+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:49.692 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:49.323+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:49.692 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:49.324+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:50.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:49.690+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:50.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:49.690+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:50.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:49.690+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:50.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:49.691+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:50.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:50.312+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:50.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:50.312+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:50.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:50.312+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:50.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:50.313+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:50.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:50.069+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:50.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:50.069+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:50.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:50.069+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:50.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:50.070+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:51.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:17:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:17:50.697+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 27486 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:17:51.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:50.671+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:51.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:50.671+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:51.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:50.671+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:51.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:50.671+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:51.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:51.094+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:51.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:51.094+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:51.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:51.094+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:51.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:51.095+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:51.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:51.334+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:51.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:51.334+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:51.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:51.334+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:51.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:51.335+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:52.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:51.662+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:52.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:51.662+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:52.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:51.662+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:52.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:51.663+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:52.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:52.113+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:52.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:52.113+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:52.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:52.113+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:52.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:52.114+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:52.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:52.289+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:52.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:52.289+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:52.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:52.289+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:52.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:52.290+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:53.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:52.696+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:53.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:52.696+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:53.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:52.696+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:53.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:52.697+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:53.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:53.282+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:53.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:53.282+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:53.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:53.282+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:53.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:53.284+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:53.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:53.108+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:53.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:53.108+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:53.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:53.108+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:53.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:53.109+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:54.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:53.710+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:54.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:53.710+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:54.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:53.710+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:54.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:53.711+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:54.331 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:54.074+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:54.331 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:54.074+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:54.332 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:54.074+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:54.332 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:54.074+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:54.724 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:54.331+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:54.724 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:54.331+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:54.724 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:54.331+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:54.724 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:54.332+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:55.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:54.723+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:55.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:54.723+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:55.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:54.723+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:55.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:54.724+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:55.377 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:55.054+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:55.377 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:55.054+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:55.377 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:55.054+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:55.377 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:55.055+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:55.664 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:55.376+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:55.664 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:55.376+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:55.664 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:55.376+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:55.664 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:55.378+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:56.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:17:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:17:55.700+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 27575 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:17:56.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:55.703+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:56.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:55.703+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:56.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:55.703+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:56.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:55.704+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:56.330 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:56.049+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:56.330 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:56.049+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:56.330 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:56.049+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:56.330 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:56.050+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:56.669 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:56.329+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:56.669 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:56.329+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:56.669 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:56.329+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:56.669 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:56.330+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:57.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:56.668+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:57.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:56.668+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:57.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:56.668+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:57.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:56.669+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:57.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:57.334+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:57.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:57.334+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:57.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:57.334+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:57.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:57.334+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:57.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:57.090+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:57.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:57.090+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:57.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:57.090+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:57.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:57.091+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:58.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:57.710+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:58.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:57.710+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:58.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:57.710+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:58.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:57.712+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:58.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:58.296+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:58.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:58.296+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:58.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:58.296+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:58.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:58.296+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:58.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:58.076+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:58.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:58.076+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:58.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:58.076+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:58.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:58.077+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:59.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:58.729+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:59.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:58.729+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:59.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:58.729+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:17:59.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:58.730+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:59.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:59.264+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:59.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:59.264+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:59.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:59.264+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:17:59.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:17:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:17:59.265+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:17:59.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:59.118+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:59.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:59.118+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:59.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:59.118+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:17:59.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:17:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:17:59.119+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:00.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:59.683+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:00.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:59.683+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:00.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:59.683+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:00.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:17:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:17:59.684+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:00.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:00.304+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:00.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:00.304+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:00.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:00.304+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:00.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:00.306+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:00.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:00.114+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:00.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:00.114+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:00.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:00.114+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:00.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:00.115+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:01.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:18:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:18:00.704+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 27664 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:18:01.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:00.733+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:01.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:00.733+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:01.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:00.733+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:01.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:00.734+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:01.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:01.290+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:01.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:01.290+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:01.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:01.290+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:01.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:01.290+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:01.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:01.118+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:01.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:01.118+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:01.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:01.118+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:01.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:01.119+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:02.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:01.774+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:02.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:01.774+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:02.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:01.774+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:02.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:01.774+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:02.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:02.278+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:02.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:02.278+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:02.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:02.278+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:02.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:02.279+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:02.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:02.087+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:02.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:02.087+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:02.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:02.087+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:02.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:02.087+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:03.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:03.050+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:03.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:03.050+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:03.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:03.050+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:03.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:03.050+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:03.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:02.810+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:03.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:02.810+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:03.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:02.810+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:03.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:02.811+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:03.624 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:03.321+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:03.624 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:03.321+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:03.625 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:03.321+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:03.625 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:03.322+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:04.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:03.859+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:04.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:03.859+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:04.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:03.859+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:04.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:03.860+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:04.357 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:04.076+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:04.357 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:04.076+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:04.357 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:04.076+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:04.357 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:04.077+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:04.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:04.356+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:04.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:04.356+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:04.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:04.356+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:04.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:04.357+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:05.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:05.107+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:05.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:05.107+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:05.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:05.107+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:05.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:05.107+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:05.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:04.881+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:05.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:04.881+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:05.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:04.881+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:05.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:04.882+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:05.707 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:05.321+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:05.708 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:05.321+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:05.708 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:05.321+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:05.708 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:05.322+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:06.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:18:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:18:05.707+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 27753 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:18:06.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:05.916+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:06.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:05.916+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:06.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:05.916+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:06.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:05.917+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:06.364 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:06.085+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:06.364 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:06.085+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:06.364 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:06.085+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:06.364 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:06.086+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:06.774 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:06.363+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:06.774 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:06.363+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:06.774 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:06.363+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:06.774 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:06.364+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:07.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:06.882+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:07.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:06.882+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:07.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:06.882+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:07.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:06.882+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:07.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:07.346+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:07.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:07.346+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:07.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:07.346+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:07.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:07.346+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:07.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:07.099+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:07.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:07.099+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:07.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:07.099+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:07.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:07.099+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:08.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:07.915+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:08.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:07.915+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:08.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:07.915+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:08.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:07.916+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:08.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:08.082+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:08.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:08.082+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:08.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:08.082+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:08.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:08.082+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:08.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:08.364+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:08.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:08.364+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:08.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:08.364+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:08.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:08.365+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:09.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:08.920+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:09.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:08.920+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:09.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:08.920+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:09.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:08.921+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:09.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:09.095+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:09.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:09.095+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:09.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:09.095+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:09.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:09.096+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:09.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:09.389+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:09.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:09.389+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:09.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:09.389+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:09.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:09.390+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:10.131 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:09.926+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:10.132 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:09.926+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:10.132 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:09.926+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:10.132 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:09.927+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:10.440 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:10.131+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:10.441 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:10.131+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:10.441 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:10.131+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:10.441 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:10.132+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:10.710 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:10.440+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:10.710 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:10.440+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:10.710 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:10.440+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:10.710 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:10.441+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:11.024 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:18:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:18:10.710+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 27841 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:18:11.024 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:10.894+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:11.024 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:10.894+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:11.025 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:10.894+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:11.025 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:10.894+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:11.281 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:11.114+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:11.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:11.114+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:11.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:11.114+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:11.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:11.115+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:11.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:11.427+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:11.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:11.427+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:11.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:11.427+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:11.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:11.427+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:12.138 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:11.936+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:12.138 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:11.936+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:12.138 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:11.936+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:12.138 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:11.937+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:12.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:12.378+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:12.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:12.378+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:12.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:12.378+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:12.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:12.379+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:12.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:12.138+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:12.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:12.138+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:12.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:12.138+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:12.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:12.139+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:13.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:13.145+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:13.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:13.145+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:13.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:13.145+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:13.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:13.146+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:13.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:12.898+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:13.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:12.898+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:13.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:12.898+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:13.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:12.899+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:13.625 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:13.338+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:13.625 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:13.338+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:13.625 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:13.338+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:13.625 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:13.339+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:14.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:14.170+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:14.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:14.170+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:14.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:14.170+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:14.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:14.171+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:14.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:13.930+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:14.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:13.930+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:14.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:13.930+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:14.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:13.930+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:14.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:14.357+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:14.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:14.357+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:14.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:14.357+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:14.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:14.357+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:15.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:15.153+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:15.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:15.153+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:15.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:15.153+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:15.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:15.153+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:15.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:14.929+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:15.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:14.929+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:15.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:14.929+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:15.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:14.930+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:15.714 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:15.323+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:15.714 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:15.323+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:15.714 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:15.323+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:15.714 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:15.324+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:16.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:18:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:18:15.713+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 27931 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:18:16.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:15.906+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:16.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:15.907+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:16.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:15.907+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:16.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:15.908+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:16.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:16.173+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:16.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:16.173+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:16.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:16.173+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:16.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:16.174+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:16.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:16.289+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:16.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:16.289+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:16.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:16.289+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:16.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:16.290+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:17.210 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:16.903+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:17.210 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:16.903+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:17.211 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:16.903+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:17.211 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:16.904+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:17.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:17.210+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:17.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:17.210+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:17.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:17.210+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:17.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:17.211+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:17.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:17.269+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:17.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:17.269+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:17.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:17.269+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:17.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:17.270+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:18.236 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:17.899+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:18.236 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:17.899+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:18.236 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:17.899+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:18.237 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:17.899+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:18.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:18.294+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:18.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:18.294+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:18.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:18.294+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:18.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:18.295+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:18.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:18.235+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:18.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:18.235+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:18.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:18.235+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:18.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:18.236+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:19.215 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:18.929+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:19.215 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:18.929+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:19.215 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:18.929+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:19.216 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:18.930+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:19.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:19.215+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:19.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:19.215+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:19.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:19.215+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:19.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:19.215+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:19.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:19.299+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:19.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:19.299+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:19.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:19.299+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:19.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:19.300+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:20.184 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:19.921+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:20.185 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:19.921+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:20.185 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:19.921+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:20.185 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:19.922+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:20.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:20.317+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:20.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:20.317+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:20.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:20.317+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:20.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:20.318+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:20.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:20.184+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:20.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:20.184+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:20.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:20.184+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:20.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:20.185+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:21.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:18:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:18:20.717+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 28019 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:18:21.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:20.887+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:21.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:20.887+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:21.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:20.887+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:21.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:20.888+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:21.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:21.304+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:21.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:21.304+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:21.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:21.304+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:21.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:21.304+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:21.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:21.154+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:21.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:21.154+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:21.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:21.154+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:21.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:21.155+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:22.181 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:21.851+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:22.181 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:21.851+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:22.181 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:21.851+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:22.181 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:21.852+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:22.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:22.297+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:22.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:22.297+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:22.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:22.297+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:22.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:22.298+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:22.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:22.180+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:22.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:22.180+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:22.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:22.180+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:22.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:22.181+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:23.187 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:22.848+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:23.188 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:22.848+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:23.188 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:22.848+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:23.188 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:22.849+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:23.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:23.269+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:23.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:23.269+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:23.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:23.269+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:23.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:23.270+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:23.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:23.187+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:23.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:23.187+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:23.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:23.187+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:23.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:23.188+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:24.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:23.839+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:24.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:23.839+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:24.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:23.839+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:24.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:23.840+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:24.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:24.266+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:24.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:24.266+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:24.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:24.266+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:24.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:24.267+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:24.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:24.142+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:24.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:24.142+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:24.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:24.142+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:24.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:24.142+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:25.155 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:24.828+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:25.155 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:24.828+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:25.156 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:24.828+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:25.156 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:24.829+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:25.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:25.276+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:25.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:25.276+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:25.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:25.276+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:25.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:25.277+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:25.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:25.154+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:25.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:25.154+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:25.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:25.154+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:25.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:25.155+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:26.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:18:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:18:25.720+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 28110 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:18:26.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:25.849+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:26.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:25.849+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:26.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:25.849+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:26.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:25.850+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:26.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:26.300+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:26.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:26.300+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:26.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:26.300+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:26.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:26.301+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:26.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:26.141+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:26.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:26.141+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:26.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:26.141+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:26.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:26.142+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:27.268 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:27.118+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:27.268 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:27.118+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:27.268 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:27.118+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:27.268 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:27.119+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:27.269 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:26.890+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:27.269 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:26.890+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:27.269 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:26.890+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:27.269 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:26.891+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:27.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:27.267+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:27.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:27.267+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:27.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:27.267+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:27.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:27.268+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:28.167 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:27.899+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:28.168 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:27.899+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:28.168 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:27.899+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:28.168 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:27.900+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:28.270+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:28.270+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:28.270+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:28.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:28.271+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:28.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:28.167+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:28.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:28.167+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:28.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:28.167+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:28.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:28.168+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:29.209 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:28.923+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:29.209 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:28.923+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:29.209 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:28.923+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:29.210 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:28.924+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:29.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:29.296+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:29.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:29.296+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:29.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:29.296+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:29.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:29.296+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:29.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:29.209+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:29.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:29.209+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:29.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:29.209+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:29.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:29.209+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:30.196 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:29.923+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:30.197 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:29.923+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:30.197 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:29.923+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:30.197 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:29.924+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:30.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:30.201+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:30.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:30.201+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:30.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:30.201+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:30.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:30.202+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:30.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:30.289+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:30.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:30.289+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:30.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:30.289+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:30.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:30.290+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:31.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:18:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:18:30.723+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 28197 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:18:31.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:30.931+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:31.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:30.931+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:31.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:30.931+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:31.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:30.931+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:31.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:31.325+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:31.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:31.325+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:31.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:31.325+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:31.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:31.325+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:31.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:31.167+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:31.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:31.167+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:31.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:31.167+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:31.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:31.168+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:32.218 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:31.937+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:32.218 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:31.937+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:32.218 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:31.937+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:32.218 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:31.937+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:32.528 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:32.371+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:32.529 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:32.371+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:32.529 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:32.371+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:32.529 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:32.372+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:32.529 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:32.217+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:32.529 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:32.217+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:32.529 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:32.217+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:32.529 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:32.218+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:33.174 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:32.923+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:33.174 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:32.923+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:33.174 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:32.923+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:33.175 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:32.924+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:33.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:33.174+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:33.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:33.174+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:33.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:33.174+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:33.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:33.176+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:33.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:33.398+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:33.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:33.398+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:33.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:33.398+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:33.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:33.399+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:34.203 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:33.883+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:34.203 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:33.883+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:34.203 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:33.883+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:34.203 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:33.884+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:34.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:34.399+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:34.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:34.399+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:34.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:34.399+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:34.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:34.400+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:34.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:34.203+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:34.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:34.203+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:34.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:34.203+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:34.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:34.204+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:35.219 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:34.875+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:35.220 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:34.875+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:35.220 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:34.875+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:35.220 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:34.876+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:35.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:35.220+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:35.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:35.220+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:35.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:35.220+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:35.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:35.221+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:35.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:35.429+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:35.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:35.429+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:35.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:35.429+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:35.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:35.429+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:35.886 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:18:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:18:35.727+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 28287 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:18:36.248 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:35.887+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:36.248 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:35.887+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:36.248 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:35.887+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:36.248 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:35.888+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:36.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:36.403+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:36.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:36.403+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:36.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:36.403+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:36.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:36.403+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:36.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:36.248+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:36.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:36.248+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:36.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:36.248+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:36.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:36.249+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:37.246 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:36.902+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:37.247 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:36.903+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:37.247 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:36.903+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:37.247 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:36.903+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:37.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:37.448+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:37.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:37.448+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:37.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:37.448+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:37.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:37.449+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:37.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:37.247+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:37.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:37.247+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:37.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:37.247+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:37.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:37.248+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:38.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:37.858+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:38.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:37.858+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:38.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:37.858+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:38.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:37.859+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:38.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:38.446+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:38.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:38.446+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:38.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:38.446+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:38.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:38.447+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:38.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:38.290+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:38.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:38.290+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:38.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:38.290+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:38.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:38.291+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:39.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:38.817+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:39.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:38.817+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:39.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:38.817+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:39.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:38.818+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:39.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:39.339+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:39.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:39.339+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:39.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:39.339+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:39.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:39.339+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:39.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:39.404+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:39.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:39.404+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:39.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:39.404+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:39.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:39.405+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:40.197 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:39.842+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:40.198 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:39.842+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:40.198 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:39.842+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:40.198 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:39.843+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:40.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:40.379+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:40.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:40.379+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:40.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:40.379+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:40.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:40.380+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:40.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:40.402+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:40.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:40.402+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:40.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:40.402+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:40.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:40.403+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:41.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:18:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:18:40.730+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 28374 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:18:41.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:40.888+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:41.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:40.888+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:41.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:40.888+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:41.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:40.889+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:41.744 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:41.395+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:41.744 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:41.395+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:41.745 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:41.395+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:41.745 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:41.395+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:41.745 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:41.447+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:41.745 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:41.447+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:41.745 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:41.447+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:41.745 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:41.448+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:42.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:41.847+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:42.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:41.847+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:42.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:41.847+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:42.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:41.847+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:42.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:42.471+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:42.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:42.471+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:42.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:42.471+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:42.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:42.472+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:42.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:42.442+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:42.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:42.442+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:42.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:42.442+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:42.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:42.443+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:43.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:42.871+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:43.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:42.871+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:43.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:42.871+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:43.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:42.872+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:43.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:43.430+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:43.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:43.430+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:43.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:43.430+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:43.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:43.431+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:43.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:43.475+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:43.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:43.475+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:43.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:43.475+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:43.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:43.477+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:44.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:43.834+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:44.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:43.834+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:44.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:43.834+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:44.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:43.835+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:44.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:44.437+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:44.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:44.437+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:44.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:44.437+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:44.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:44.438+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:44.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:44.405+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:44.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:44.405+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:44.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:44.406+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:44.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:44.407+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:45.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:44.804+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:45.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:44.804+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:45.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:44.804+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:45.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:44.805+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:45.733 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:45.441+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:45.734 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:45.442+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:45.734 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:45.442+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:45.734 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:45.442+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:45.734 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:45.420+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:45.734 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:45.420+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:45.734 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:45.420+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:45.735 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:45.421+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:46.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:18:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:18:45.733+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 28465 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:18:46.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:45.784+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:46.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:45.784+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:46.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:45.784+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:46.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:45.786+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:46.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:46.384+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:46.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:46.384+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:46.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:46.384+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:46.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:46.385+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:46.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:46.423+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:46.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:46.423+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:46.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:46.423+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:46.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:46.423+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:47.125 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:46.828+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:47.125 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:46.828+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:47.125 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:46.828+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:47.126 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:46.829+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:47.458 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:47.367+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:47.458 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:47.367+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:47.458 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:47.367+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:47.458 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:47.368+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:47.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:47.457+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:47.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:47.457+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:47.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:47.457+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:47.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:47.458+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:48.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:47.852+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:48.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:47.852+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:48.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:47.852+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:48.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:47.853+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:48.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:48.430+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:48.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:48.430+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:48.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:48.430+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:48.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:48.431+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:48.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:48.368+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:48.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:48.368+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:48.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:48.368+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:48.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:48.369+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:49.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:48.814+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:49.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:48.814+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:49.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:48.814+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:49.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:48.815+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:49.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:49.436+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:49.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:49.436+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:49.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:49.436+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:49.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:49.437+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:49.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:49.377+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:49.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:49.377+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:49.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:49.377+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:49.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:49.378+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:50.198 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:49.795+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:50.198 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:49.795+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:50.198 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:49.795+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:50.198 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:49.796+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:50.466 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:50.348+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:50.466 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:50.348+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:50.466 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:50.348+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:50.467 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:50.348+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:50.736 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:50.466+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:50.737 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:50.466+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:50.737 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:50.466+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:50.737 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:50.467+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:51.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:18:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:18:50.736+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 28553 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:18:51.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:50.759+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:51.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:50.759+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:51.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:50.759+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:51.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:50.760+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:51.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:51.486+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:51.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:51.486+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:51.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:51.486+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:51.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:51.486+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:51.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:51.354+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:51.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:51.354+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:51.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:51.354+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:51.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:51.354+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:52.170 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:51.803+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:52.170 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:51.803+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:52.171 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:51.803+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:52.171 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:51.804+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:52.472 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:52.334+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:52.472 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:52.334+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:52.472 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:52.334+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:52.472 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:52.335+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:52.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:52.471+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:52.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:52.471+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:52.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:52.471+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:52.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:52.472+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:53.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:52.797+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:53.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:52.797+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:53.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:52.797+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:53.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:52.798+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:53.627 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:53.357+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:53.628 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:53.357+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:53.628 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:53.357+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:53.628 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:53.357+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:53.628 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:53.482+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:53.628 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:53.482+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:53.628 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:53.482+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:53.628 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:53.483+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:54.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:53.777+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:54.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:53.777+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:54.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:53.777+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:54.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:53.778+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:54.765 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:54.493+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:54.766 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:54.493+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:54.766 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:54.493+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:54.766 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:54.493+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:54.766 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:54.382+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:54.767 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:54.382+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:54.767 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:54.382+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:54.767 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:54.382+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:55.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:54.765+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:55.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:54.765+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:55.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:54.765+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:55.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:54.766+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:55.740 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:55.513+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:55.740 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:55.513+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:55.740 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:55.513+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:55.741 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:55.514+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:55.741 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:55.376+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:55.741 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:55.376+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:55.741 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:55.376+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:55.741 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:55.376+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:56.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:18:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:18:55.739+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 28643 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:18:56.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:55.783+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:56.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:55.783+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:56.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:55.783+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:56.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:55.783+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:56.759 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:56.409+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:56.759 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:56.409+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:56.759 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:56.409+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:56.759 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:56.410+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:56.759 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:56.501+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:56.760 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:56.501+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:56.760 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:56.501+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:56.760 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:56.502+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:57.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:56.758+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:57.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:56.758+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:57.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:56.758+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:57.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:56.759+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:57.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:57.417+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:57.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:57.417+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:57.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:57.417+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:57.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:57.418+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:57.911 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:57.534+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:57.911 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:57.534+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:57.911 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:57.534+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:57.911 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:57.534+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:57.912 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:57.765+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:57.912 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:57.765+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:57.912 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:57.765+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:57.912 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:57.766+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:58.497 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:58.449+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:58.498 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:58.449+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:58.498 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:58.449+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:58.498 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:58.450+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:58.498 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:58.496+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:58.498 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:58.496+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:58.498 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:58.496+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:58.499 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:58.497+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:58.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:58.720+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:58.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:58.720+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:58.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:58.720+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:18:58.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:58.721+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:59.716 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:59.489+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:59.716 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:59.489+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:59.716 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:59.489+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:18:59.717 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:18:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:18:59.490+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:18:59.717 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:59.429+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:59.717 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:59.429+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:59.717 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:59.429+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:18:59.717 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:18:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:18:59.430+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:00.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:59.715+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:00.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:59.715+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:00.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:59.715+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:00.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:18:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:18:59.716+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:00.530 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:00.385+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:00.530 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:00.385+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:00.530 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:00.385+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:00.530 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:00.386+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:00.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:19:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:19:00.743+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 28731 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:19:00.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:00.670+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:00.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:00.670+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:00.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:00.670+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:00.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:00.670+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:00.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:00.529+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:00.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:00.529+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:00.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:00.529+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:00.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:00.530+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:01.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:01.509+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:01.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:01.509+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:01.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:01.509+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:01.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:01.510+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:01.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:01.655+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:01.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:01.655+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:01.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:01.655+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:01.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:01.655+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:01.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:01.414+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:01.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:01.414+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:01.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:01.414+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:01.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:01.415+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:02.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:02.453+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:02.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:02.453+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:02.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:02.453+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:02.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:02.454+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:03.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:02.699+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:03.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:02.699+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:03.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:02.699+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:03.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:02.700+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:03.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:02.553+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:03.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:02.553+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:03.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:02.553+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:03.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:02.554+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:03.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:03.467+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:03.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:03.467+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:03.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:03.467+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:03.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:03.468+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:04.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:03.657+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:04.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:03.657+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:04.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:03.657+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:04.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:03.658+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:04.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:03.585+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:04.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:03.585+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:04.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:03.585+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:04.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:03.586+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:04.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:04.654+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:04.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:04.654+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:04.785 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:04.654+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:04.785 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:04.655+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:04.785 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:04.572+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:04.786 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:04.572+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:04.786 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:04.572+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:04.786 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:04.572+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:04.786 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:04.425+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:04.787 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:04.425+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:04.787 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:04.425+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:04.787 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:04.426+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:05.747 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:05.612+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:05.747 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:05.612+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:05.747 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:05.612+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:05.748 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:05.612+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:05.748 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:05.684+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:05.748 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:05.684+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:05.748 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:05.684+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:05.748 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:05.685+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:05.749 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:05.438+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:05.749 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:05.438+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:05.749 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:05.438+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:05.749 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:05.439+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:06.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:19:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:19:05.746+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 28822 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:19:06.571 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:06.488+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:06.572 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:06.488+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:06.572 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:06.488+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:06.572 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:06.489+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:07.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:06.640+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:07.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:06.640+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:07.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:06.640+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:07.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:06.641+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:07.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:06.571+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:07.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:06.571+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:07.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:06.571+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:07.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:06.572+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:07.553 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:07.533+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:07.553 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:07.533+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:07.554 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:07.533+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:07.554 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:07.534+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:08.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:07.553+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:08.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:07.553+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:08.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:07.553+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:08.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:07.554+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:08.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:07.646+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:08.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:07.646+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:08.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:07.646+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:08.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:07.647+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:08.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:08.661+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:08.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:08.661+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:08.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:08.661+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:08.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:08.661+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:08.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:08.531+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:08.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:08.531+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:08.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:08.531+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:08.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:08.531+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:08.785 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:08.547+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:08.785 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:08.547+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:08.786 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:08.547+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:08.786 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:08.547+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:09.802 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:09.536+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:09.802 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:09.536+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:09.802 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:09.536+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:09.803 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:09.537+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:09.803 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:09.563+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:09.803 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:09.563+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:09.803 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:09.563+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:09.803 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:09.564+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:09.804 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:09.666+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:09.804 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:09.666+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:09.804 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:09.666+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:09.804 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:09.666+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:10.763 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:19:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:19:10.750+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 28910 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:19:10.763 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:10.580+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:10.763 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:10.580+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:10.763 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:10.581+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:10.764 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:10.581+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:10.764 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:10.508+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:10.764 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:10.508+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:10.764 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:10.508+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:10.764 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:10.509+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:10.765 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:10.698+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:10.765 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:10.698+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:10.765 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:10.698+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:10.765 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:10.699+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:11.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:11.525+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:11.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:11.525+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:11.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:11.525+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:11.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:11.526+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:11.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:11.721+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:11.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:11.721+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:11.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:11.721+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:11.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:11.722+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:11.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:11.569+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:11.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:11.569+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:11.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:11.569+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:11.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:11.570+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:12.616 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:12.541+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:12.616 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:12.541+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:12.616 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:12.541+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:12.617 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:12.542+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:13.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:12.673+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:13.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:12.673+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:13.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:12.673+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:13.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:12.674+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:13.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:12.616+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:13.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:12.616+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:13.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:12.616+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:13.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:12.617+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:14.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:13.647+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:14.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:13.647+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:14.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:13.647+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:14.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:13.648+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:14.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:13.559+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:14.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:13.559+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:14.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:13.559+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:14.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:13.560+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:14.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:13.666+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:14.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:13.666+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:14.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:13.666+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:14.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:13.667+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:15.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:14.660+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:15.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:14.660+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:15.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:14.660+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:15.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:14.661+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:15.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:14.558+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:15.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:14.558+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:15.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:14.558+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:15.034 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:14.559+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:15.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:14.622+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:15.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:14.622+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:15.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:14.622+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:15.035 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:14.623+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:15.878 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:19:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:19:15.753+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 28998 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:19:15.879 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:15.583+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:15.879 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:15.583+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:15.879 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:15.583+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:15.879 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:15.584+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:15.879 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:15.707+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:15.879 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:15.707+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:15.880 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:15.707+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:15.880 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:15.708+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:15.880 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:15.574+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:15.880 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:15.574+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:15.880 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:15.574+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:15.880 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:15.575+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:17.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:16.579+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:17.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:16.579+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:17.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:16.579+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:17.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:16.580+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:17.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:16.668+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:17.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:16.668+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:17.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:16.668+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:17.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:16.669+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:17.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:16.585+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:17.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:16.585+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:17.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:16.585+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:17.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:16.586+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:17.629 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:17.553+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:17.630 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:17.553+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:17.630 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:17.553+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:17.630 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:17.554+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:17.630 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:17.553+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:17.630 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:17.553+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:17.630 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:17.553+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:17.631 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:17.554+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:18.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:17.629+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:18.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:17.629+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:18.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:17.629+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:18.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:17.630+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:19.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:18.571+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:19.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:18.571+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:19.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:18.571+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:19.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:18.571+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:19.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:18.538+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:19.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:18.538+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:19.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:18.538+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:19.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:18.539+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:19.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:18.612+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:19.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:18.612+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:19.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:18.612+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:19.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:18.612+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:19.855 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:19.572+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:19.855 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:19.572+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:19.855 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:19.572+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:19.855 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:19.573+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:19.855 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:19.565+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:19.855 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:19.565+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:19.856 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:19.565+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:19.856 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:19.565+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:19.856 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:19.605+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:19.856 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:19.605+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:19.856 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:19.606+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:19.856 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:19.606+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:20.880 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:19:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:19:20.757+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 29084 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:19:20.881 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:20.566+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:20.881 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:20.566+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:20.881 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:20.566+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:20.881 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:20.566+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:20.882 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:20.531+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:20.882 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:20.531+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:20.882 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:20.531+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:20.882 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:20.532+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:20.882 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:20.568+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:20.882 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:20.568+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:20.882 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:20.568+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:20.883 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:20.569+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:21.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:21.539+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:21.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:21.539+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:21.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:21.539+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:21.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:21.540+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:21.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:21.525+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:21.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:21.525+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:21.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:21.525+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:21.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:21.526+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:21.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:21.555+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:21.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:21.555+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:21.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:21.555+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:21.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:21.556+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:22.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:22.494+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:22.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:22.494+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:22.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:22.494+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:22.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:22.494+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:22.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:22.550+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:22.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:22.550+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:22.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:22.550+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:22.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:22.551+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:22.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:22.502+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:22.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:22.502+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:22.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:22.502+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:22.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:22.503+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:23.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:23.459+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:23.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:23.459+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:23.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:23.459+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:23.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:23.460+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:23.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:23.556+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:23.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:23.557+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:23.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:23.557+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:23.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:23.557+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:23.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:23.511+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:23.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:23.511+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:23.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:23.511+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:23.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:23.512+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:24.752 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:24.421+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:24.752 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:24.421+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:24.753 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:24.421+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:24.753 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:24.421+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:24.753 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:24.508+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:24.753 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:24.508+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:24.753 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:24.508+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:24.753 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:24.509+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:24.754 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:24.497+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:24.754 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:24.497+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:24.754 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:24.497+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:24.754 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:24.498+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:25.761 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:25.530+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:25.762 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:25.530+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:25.762 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:25.530+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:25.762 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:25.531+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:25.762 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:25.457+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:25.763 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:25.457+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:25.763 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:25.457+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:25.763 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:25.458+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:25.763 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:25.454+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:25.763 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:25.454+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:25.763 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:25.454+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:25.764 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:25.454+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:26.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:19:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:19:25.760+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 29175 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:19:26.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:26.492+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:26.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:26.492+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:26.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:26.492+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:26.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:26.492+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:26.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:26.534+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:26.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:26.534+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:26.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:26.534+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:26.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:26.535+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:26.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:26.481+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:26.785 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:26.481+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:26.785 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:26.481+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:26.785 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:26.482+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:27.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:27.501+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:27.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:27.501+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:27.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:27.501+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:27.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:27.502+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:27.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:27.472+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:27.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:27.472+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:27.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:27.472+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:27.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:27.473+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:27.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:27.557+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:27.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:27.557+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:27.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:27.557+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:27.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:27.557+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:28.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:28.479+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:28.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:28.479+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:28.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:28.479+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:28.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:28.480+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:28.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:28.574+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:28.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:28.574+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:28.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:28.574+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:28.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:28.575+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:28.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:28.498+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:28.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:28.498+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:28.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:28.498+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:28.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:28.498+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:29.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:29.518+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:29.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:29.518+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:29.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:29.518+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:29.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:29.519+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:29.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:29.615+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:29.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:29.615+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:29.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:29.615+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:29.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:29.616+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:29.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:29.519+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:29.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:29.519+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:29.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:29.519+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:29.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:29.521+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:30.764 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:30.589+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:30.764 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:30.589+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:30.764 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:30.589+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:30.765 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:30.591+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:30.765 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:30.512+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:30.765 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:30.512+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:30.765 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:30.512+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:30.765 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:30.512+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:30.766 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:30.493+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:30.766 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:30.493+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:30.766 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:30.493+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:30.766 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:30.495+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:31.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:19:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:19:30.763+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 29262 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:19:31.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:31.482+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:31.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:31.482+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:31.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:31.482+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:31.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:31.483+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:31.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:31.521+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:31.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:31.521+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:31.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:31.521+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:31.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:31.522+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:31.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:31.557+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:31.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:31.557+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:31.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:31.557+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:31.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:31.558+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:32.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:32.517+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:32.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:32.517+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:32.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:32.517+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:32.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:32.517+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:32.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:32.546+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:32.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:32.546+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:32.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:32.546+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:32.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:32.547+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:32.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:32.546+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:32.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:32.546+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:32.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:32.546+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:32.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:32.547+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:33.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:33.513+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:33.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:33.513+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:33.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:33.513+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:33.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:33.513+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:33.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:33.509+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:33.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:33.509+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:33.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:33.509+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:33.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:33.510+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:33.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:33.524+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:33.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:33.524+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:33.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:33.524+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:33.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:33.525+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:34.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:34.503+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:34.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:34.503+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:34.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:34.503+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:34.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:34.504+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:34.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:34.538+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:34.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:34.538+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:34.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:34.538+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:34.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:34.538+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:34.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:34.553+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:34.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:34.553+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:34.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:34.553+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:34.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:34.553+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:35.767 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:35.571+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:35.768 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:35.571+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:35.768 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:35.571+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:35.768 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:35.572+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:35.768 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:35.535+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:35.768 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:35.535+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:35.769 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:35.535+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:35.769 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:35.536+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:35.769 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:35.499+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:35.769 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:35.499+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:35.769 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:35.499+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:35.769 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:35.499+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:36.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:19:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:19:35.767+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 29352 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:19:36.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:36.576+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:36.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:36.576+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:36.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:36.576+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:36.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:36.577+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:36.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:36.547+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:36.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:36.547+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:36.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:36.547+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:36.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:36.547+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:36.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:36.500+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:36.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:36.500+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:36.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:36.500+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:36.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:36.501+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:37.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:37.541+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:37.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:37.541+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:37.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:37.541+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:37.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:37.543+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:37.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:37.616+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:37.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:37.616+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:37.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:37.616+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:37.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:37.617+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:37.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:37.528+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:37.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:37.528+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:37.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:37.528+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:37.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:37.528+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:38.633+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:38.633+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:38.633+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:39.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:38.634+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:39.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:38.552+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:39.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:38.552+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:39.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:38.552+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:39.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:38.552+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:39.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:38.539+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:39.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:38.539+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:39.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:38.539+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:39.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:38.539+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:39.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:39.638+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:39.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:39.638+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:39.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:39.638+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:39.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:39.639+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:39.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:39.601+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:39.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:39.601+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:39.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:39.601+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:39.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:39.602+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:39.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:39.490+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:39.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:39.490+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:39.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:39.490+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:39.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:39.491+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:40.794 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:19:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:19:40.770+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 29440 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:19:40.795 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:40.534+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:40.795 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:40.534+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:40.795 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:40.534+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:40.795 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:40.534+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:40.795 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:40.669+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:40.796 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:40.669+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:40.796 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:40.669+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:40.796 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:40.670+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:40.796 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:40.592+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:40.796 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:40.592+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:40.796 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:40.592+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:40.796 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:40.593+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:42.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:41.635+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:42.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:41.635+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:42.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:41.635+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:42.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:41.635+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:42.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:41.605+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:42.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:41.606+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:42.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:41.606+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:42.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:41.607+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:42.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:41.543+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:42.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:41.543+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:42.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:41.543+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:42.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:41.544+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:43.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:42.642+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:43.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:42.642+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:43.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:42.642+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:43.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:42.643+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:43.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:42.635+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:43.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:42.635+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:43.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:42.635+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:43.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:42.636+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:43.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:42.591+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:43.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:42.591+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:43.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:42.591+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:43.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:42.592+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:44.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:43.589+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:44.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:43.589+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:44.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:43.589+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:44.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:43.590+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:44.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:43.667+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:44.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:43.667+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:44.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:43.667+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:44.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:43.668+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:44.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:43.584+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:44.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:43.584+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:44.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:43.584+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:44.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:43.585+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:45.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:44.691+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:45.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:44.691+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:45.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:44.691+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:45.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:44.692+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:45.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:44.602+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:45.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:44.602+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:45.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:44.602+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:45.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:44.603+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:45.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:44.557+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:45.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:44.557+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:45.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:44.557+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:45.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:44.557+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:45.896 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:19:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:19:45.774+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 29531 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:19:45.897 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:45.597+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:45.897 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:45.597+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:45.897 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:45.597+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:45.897 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:45.598+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:45.898 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:45.588+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:45.898 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:45.588+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:45.898 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:45.588+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:45.899 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:45.589+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:45.899 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:45.695+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:45.899 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:45.695+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:45.899 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:45.695+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:45.899 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:45.696+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:47.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:46.546+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:47.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:46.546+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:47.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:46.546+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:47.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:46.546+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:47.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:46.715+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:47.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:46.715+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:47.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:46.715+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:47.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:46.716+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:47.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:46.580+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:47.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:46.580+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:47.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:46.580+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:47.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:46.581+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:47.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:47.506+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:47.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:47.506+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:47.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:47.506+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:47.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:47.507+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:47.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:47.744+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:47.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:47.744+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:47.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:47.744+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:47.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:47.745+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:47.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:47.619+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:47.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:47.619+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:47.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:47.619+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:47.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:47.620+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:48.747 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:48.598+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:48.747 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:48.598+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:48.747 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:48.598+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:48.748 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:48.598+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:48.748 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:48.478+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:48.748 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:48.478+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:48.748 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:48.478+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:48.748 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:48.479+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:49.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:48.746+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:49.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:48.746+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:49.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:48.746+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:49.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:48.747+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:49.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:49.489+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:49.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:49.489+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:49.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:49.489+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:49.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:49.490+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:49.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:49.578+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:49.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:49.578+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:49.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:49.578+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:49.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:49.578+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:49.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:49.714+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:49.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:49.714+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:49.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:49.714+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:49.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:49.714+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:50.754 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:50.498+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:50.754 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:50.498+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:50.754 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:50.498+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:50.754 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:50.499+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:50.755 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:50.615+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:50.755 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:50.615+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:50.755 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:50.615+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:50.755 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:50.616+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:51.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:19:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:19:50.777+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 29619 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:19:51.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:50.753+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:51.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:50.753+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:51.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:50.753+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:51.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:50.754+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:51.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:51.756+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:51.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:51.756+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:51.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:51.756+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:51.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:51.757+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:51.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:51.633+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:51.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:51.633+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:51.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:51.633+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:51.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:51.634+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:51.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:51.511+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:51.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:51.511+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:51.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:51.511+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:51.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:51.512+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:52.657 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:52.523+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:52.657 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:52.523+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:52.657 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:52.523+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:52.657 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:52.524+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:52.657 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:52.616+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:52.658 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:52.616+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:52.658 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:52.616+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:52.658 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:52.617+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:53.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:52.765+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:53.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:52.765+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:53.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:52.765+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:53.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:52.765+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:54.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:53.566+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:54.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:53.566+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:54.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:53.566+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:54.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:53.567+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:54.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:53.743+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:54.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:53.743+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:54.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:53.743+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:54.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:53.744+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:54.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:53.609+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:54.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:53.609+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:54.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:53.609+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:54.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:53.610+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:54.781 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:54.528+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:54.781 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:54.528+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:54.781 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:54.528+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:54.781 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:54.528+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:54.781 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:54.609+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:54.781 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:54.609+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:54.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:54.609+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:54.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:54.609+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:55.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:54.781+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:55.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:54.781+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:55.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:54.781+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:55.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:54.781+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:55.781 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:55.561+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:55.781 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:55.561+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:55.781 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:55.561+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:55.781 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:55.561+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:55.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:55.544+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:55.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:55.544+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:55.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:55.544+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:55.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:55.545+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:56.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:19:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:19:55.781+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 29708 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:19:56.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:55.816+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:56.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:55.816+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:56.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:55.816+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:56.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:55.817+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:56.849 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:56.579+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:56.849 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:56.579+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:56.849 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:56.579+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:22.230699+0000 (oldest deadline 2024-03-24T10:09:42.230699+0000) 2024-03-24T10:19:56.849 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:56.580+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:56.850 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:56.565+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:56.850 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:56.565+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:56.850 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:56.565+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:56.850 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:56.566+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:57.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:56.849+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:57.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:56.849+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:57.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:56.849+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:57.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:56.850+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:58.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:57.572+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:58.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:57.581+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:58.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:57.581+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:58.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:57.581+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:58.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:57.582+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:58.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:57.815+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:58.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:57.815+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:58.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:57.815+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:58.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:57.816+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:58.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:58.528+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:58.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:58.545+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:58.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:58.545+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:58.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:58.545+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:58.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:58.546+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:59.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:58.785+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:59.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:58.785+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:59.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:58.785+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:19:59.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:58.786+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:59.826 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:19:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:19:59.542+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:19:59.827 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:59.590+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:59.827 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:59.590+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:59.827 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:59.590+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:19:59.827 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:19:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:19:59.591+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:00.203 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:59.826+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:20:00.203 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:59.826+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:20:00.203 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:59.826+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:20:00.204 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:19:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:19:59.828+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:00.908 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:20:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:20:00.785+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 29796 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:20:00.908 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:00.587+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:00.909 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:00.825+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:20:00.909 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:00.825+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:20:00.909 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:00.825+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:20:00.910 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:00.826+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:00.910 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:00.605+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:20:00.910 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:00.605+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:20:00.910 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:00.605+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:20:00.911 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:00.606+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:02.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:01.582+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:02.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:01.632+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:20:02.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:01.632+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:20:02.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:01.632+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:20:02.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:01.632+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:02.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:01.776+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:20:02.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:01.776+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:20:02.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:01.776+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:20:02.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:01.777+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:02.790 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:02.534+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:02.790 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:02.664+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:20:02.791 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:02.664+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:20:02.791 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:02.664+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:20:02.791 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:02.664+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:03.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:02.790+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:20:03.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:02.790+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:20:03.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:02.790+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:20:03.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:02.791+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:04.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:03.584+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:04.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:03.663+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:20:04.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:03.663+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:20:04.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:03.663+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:20:04.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:03.663+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:04.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:03.789+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:20:04.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:03.789+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:20:04.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:03.789+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:28.136591+0000 (oldest deadline 2024-03-24T10:09:48.136591+0000) 2024-03-24T10:20:04.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:03.790+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:05.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:04.586+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:05.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:04.793+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:05.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:04.665+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:20:05.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:04.665+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:20:05.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:04.665+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:20:05.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:04.666+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:05.810 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:20:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:20:05.788+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 29885 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:20:05.811 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:05.557+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:05.811 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:05.626+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:20:05.811 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:05.626+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:20:05.812 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:05.626+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:20:05.812 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:05.626+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:06.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:05.824+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:06.842 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:06.568+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:06.842 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:06.602+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:20:06.842 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:06.602+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:20:06.842 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:06.602+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:09:31.476091+0000 (oldest deadline 2024-03-24T10:09:51.476091+0000) 2024-03-24T10:20:06.843 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:06.603+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:07.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:06.842+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:07.885 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:07.539+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:07.885 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:07.602+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:08.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:07.884+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:08.858 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:08.559+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:08.858 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:08.605+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:09.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:08.857+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:09.853 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:09.553+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:09.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:09.655+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:10.203 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:09.888+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:10.792 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:10.537+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:10.792 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:10.699+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:11.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:20:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:20:10.791+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 29974 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:20:11.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:10.874+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:11.858 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:11.704+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:11.859 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:11.553+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:12.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:11.858+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:12.818 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:12.548+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:12.819 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:12.713+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:13.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:12.895+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:13.904 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:13.556+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:13.905 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:13.747+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:14.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:13.935+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:14.917 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:14.563+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:14.917 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:14.763+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:15.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:14.917+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:15.917 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:20:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:20:15.794+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 30064 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:20:15.917 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:15.598+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:15.917 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:15.811+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:16.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:15.927+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:16.946 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:16.596+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:16.946 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:16.821+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:17.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:16.945+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:17.919 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:17.617+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:17.919 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:17.859+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:18.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:17.918+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:18.918 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:18.630+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:18.918 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:18.824+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:19.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:18.917+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:19.951 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:19.615+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:19.952 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:19.838+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:20.204 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:19.951+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:20.868 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:20:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:20:20.798+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 30152 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:20:20.869 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:20.574+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:20.869 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:20.574+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:20.869 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:20.574+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:20.869 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:20.575+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:21.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:20.921+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:21.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:20.868+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:21.867 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:21.533+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:21.868 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:21.533+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:21.868 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:21.533+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:21.868 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:21.534+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:22.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:21.964+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:22.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:21.867+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:22.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:22.501+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:22.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:22.501+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:22.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:22.501+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:22.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:22.501+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:23.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:22.965+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:23.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:22.889+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:23.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:23.525+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:23.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:23.525+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:23.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:23.525+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:23.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:23.526+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:24.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:23.986+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:24.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:23.938+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:24.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:24.529+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:24.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:24.529+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:24.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:24.529+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:24.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:24.530+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:25.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:24.977+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:25.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:25.036+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:25.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:25.509+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:25.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:25.509+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:25.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:25.509+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:25.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:25.510+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:26.192 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:20:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:20:25.801+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 30243 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:20:26.193 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:25.968+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:26.193 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:26.037+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:26.194 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:26.037+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:26.194 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:26.037+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:26.194 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:26.037+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:26.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:26.514+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:26.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:26.514+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:26.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:26.514+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:26.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:26.515+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:27.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:27.013+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:27.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:27.072+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:27.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:27.072+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:27.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:27.072+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:27.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:27.073+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:27.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:27.469+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:27.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:27.469+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:27.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:27.469+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:27.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:27.470+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:28.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:27.993+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:28.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:28.075+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:28.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:28.075+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:28.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:28.075+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:28.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:28.076+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:28.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:28.506+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:28.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:28.506+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:28.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:28.506+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:28.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:28.506+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:29.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:29.013+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:29.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:29.072+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:29.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:29.072+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:29.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:29.072+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:29.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:29.072+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:29.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:29.511+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:29.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:29.511+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:29.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:29.511+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:29.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:29.512+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:30.205 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:30.055+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:30.205 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:30.055+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:30.206 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:30.055+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:30.206 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:30.056+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:30.206 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:30.083+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:30.206 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:30.083+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:30.206 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:30.083+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:30.207 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:30.084+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:30.805 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:30.535+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:30.805 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:30.535+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:30.805 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:30.535+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:30.806 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:30.536+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:31.120 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:20:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:20:30.804+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 30331 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:20:31.121 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:31.022+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:31.121 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:31.022+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:31.121 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:31.022+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:31.121 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:31.023+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:31.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:31.120+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:31.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:31.120+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:31.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:31.120+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:31.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:31.121+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:32.017 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:31.556+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:32.017 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:31.556+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:32.017 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:31.556+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:32.017 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:31.556+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:32.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:32.143+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:32.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:32.143+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:32.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:32.143+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:32.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:32.144+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:32.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:32.016+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:32.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:32.016+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:32.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:32.016+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:32.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:32.017+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:32.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:32.547+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:32.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:32.547+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:32.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:32.547+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:32.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:32.548+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:33.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:33.028+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:33.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:33.028+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:33.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:33.028+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:33.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:33.029+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:33.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:33.194+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:33.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:33.194+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:33.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:33.194+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:33.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:33.194+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:33.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:33.507+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:33.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:33.507+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:33.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:33.507+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:33.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:33.507+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:34.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:34.156+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:34.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:34.156+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:34.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:34.156+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:34.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:34.157+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:34.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:33.984+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:34.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:33.984+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:34.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:33.984+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:34.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:33.985+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:34.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:34.501+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:34.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:34.501+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:34.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:34.501+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:34.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:34.502+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:35.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:35.017+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:35.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:35.017+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:35.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:35.017+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:35.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:35.017+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:35.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:35.175+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:35.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:35.175+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:35.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:35.175+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:35.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:35.176+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:35.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:35.500+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:35.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:35.500+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:35.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:35.500+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:35.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:35.501+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:36.146 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:20:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:20:35.808+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 30420 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:20:36.147 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:36.049+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:36.147 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:36.049+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:36.147 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:36.049+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:36.147 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:36.050+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:36.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:36.146+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:36.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:36.146+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:36.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:36.146+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:36.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:36.147+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:37.008 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:36.550+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:37.009 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:36.550+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:37.009 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:36.550+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:37.009 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:36.551+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:37.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:37.008+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:37.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:37.008+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:37.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:37.008+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:37.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:37.009+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:37.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:37.193+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:37.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:37.193+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:37.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:37.193+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:37.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:37.193+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:38.013 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:37.540+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:38.014 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:37.540+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:38.014 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:37.540+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:38.014 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:37.540+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:38.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:38.220+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:38.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:38.220+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:38.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:38.220+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:38.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:38.221+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:38.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:38.013+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:38.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:38.013+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:38.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:38.013+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:38.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:38.014+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:38.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:38.515+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:38.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:38.515+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:38.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:38.515+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:38.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:38.516+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:39.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:39.236+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:39.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:39.236+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:39.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:39.236+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:39.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:39.237+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:39.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:38.995+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:39.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:38.995+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:39.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:38.995+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:39.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:38.996+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:39.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:39.524+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:39.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:39.524+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:39.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:39.524+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:39.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:39.525+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:40.205 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:39.948+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:40.206 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:39.948+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:40.206 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:39.948+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:40.206 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:39.949+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:40.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:40.235+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:40.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:40.235+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:40.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:40.235+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:40.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:40.237+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:40.821 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:20:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:20:40.811+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 30507 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:20:40.822 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:40.564+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:40.822 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:40.564+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:40.822 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:40.564+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:40.822 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:40.565+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:41.235 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:40.936+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:41.235 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:40.936+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:41.235 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:40.936+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:41.236 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:40.937+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:41.529 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:41.234+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:41.529 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:41.234+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:41.529 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:41.234+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:41.529 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:41.235+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:41.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:41.528+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:41.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:41.528+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:41.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:41.528+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:41.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:41.529+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:42.188 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:41.899+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:42.188 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:41.899+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:42.188 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:41.899+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:42.189 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:41.900+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:42.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:42.187+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:42.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:42.188+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:42.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:42.188+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:42.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:42.188+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:42.926 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:42.572+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:42.926 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:42.572+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:42.926 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:42.572+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:42.926 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:42.573+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:43.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:43.164+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:43.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:43.164+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:43.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:43.164+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:43.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:43.165+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:43.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:42.925+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:43.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:42.925+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:43.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:42.925+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:43.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:42.926+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:43.973 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:43.567+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:43.973 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:43.567+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:43.973 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:43.567+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:43.973 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:43.567+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:44.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:44.156+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:44.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:44.156+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:44.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:44.156+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:44.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:44.157+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:44.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:43.972+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:44.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:43.972+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:44.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:43.972+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:44.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:43.973+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:44.997 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:44.603+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:44.998 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:44.603+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:44.998 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:44.603+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:44.998 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:44.604+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:45.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:44.997+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:45.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:44.997+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:45.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:44.997+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:45.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:44.998+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:45.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:45.123+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:45.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:45.123+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:45.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:45.123+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:45.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:45.123+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:45.940 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:20:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:20:45.815+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 30598 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:20:45.940 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:45.648+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:45.940 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:45.648+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:45.940 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:45.648+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:45.940 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:45.649+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:46.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:45.979+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:46.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:45.979+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:46.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:45.979+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:46.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:45.980+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:46.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:46.129+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:46.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:46.129+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:46.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:46.129+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:46.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:46.131+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:46.938 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:46.650+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:46.939 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:46.650+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:46.939 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:46.650+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:46.939 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:46.651+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:47.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:46.938+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:47.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:46.938+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:47.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:46.938+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:47.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:46.939+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:47.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:47.160+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:47.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:47.160+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:47.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:47.160+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:47.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:47.161+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:48.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:47.693+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:48.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:47.693+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:48.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:47.693+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:48.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:47.693+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:48.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:47.911+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:48.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:47.911+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:48.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:47.911+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:48.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:47.912+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:48.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:48.199+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:48.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:48.199+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:48.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:48.199+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:48.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:48.200+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:48.915 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:48.647+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:48.915 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:48.647+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:48.916 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:48.647+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:48.916 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:48.649+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:49.183 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:48.915+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:49.183 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:48.915+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:49.183 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:48.915+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:49.183 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:48.916+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:49.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:49.182+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:49.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:49.182+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:49.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:49.182+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:49.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:49.183+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:50.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:49.683+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:50.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:49.683+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:50.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:49.683+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:50.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:49.683+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:50.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:49.877+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:50.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:49.877+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:50.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:49.877+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:50.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:49.878+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:50.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:50.205+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:50.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:50.205+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:50.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:50.205+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:50.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:50.205+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:50.942 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:20:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:20:50.818+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 30685 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:20:50.942 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:50.646+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:50.943 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:50.646+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:50.943 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:50.646+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:50.943 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:50.646+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:50.943 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:50.863+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:50.943 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:50.863+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:50.943 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:50.863+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:50.943 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:50.864+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:51.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:51.201+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:51.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:51.201+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:51.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:51.201+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:51.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:51.202+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:52.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:51.645+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:52.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:51.645+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:52.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:51.645+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:52.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:51.645+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:52.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:51.826+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:52.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:51.826+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:52.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:51.826+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:52.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:51.827+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:52.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:52.216+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:52.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:52.216+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:52.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:52.216+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:52.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:52.218+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:53.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:52.680+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:53.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:52.680+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:53.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:52.680+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:53.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:52.681+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:53.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:52.822+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:53.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:52.822+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:53.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:52.822+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:53.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:52.823+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:53.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:53.264+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:53.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:53.264+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:53.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:53.264+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:53.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:53.264+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:54.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:53.856+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:54.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:53.856+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:54.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:53.856+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:54.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:53.857+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:54.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:53.639+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:54.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:53.639+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:54.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:53.639+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:54.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:53.639+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:54.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:54.239+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:54.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:54.239+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:54.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:54.239+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:54.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:54.239+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:55.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:54.869+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:55.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:54.869+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:55.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:54.869+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:55.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:54.869+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:55.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:54.651+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:55.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:54.651+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:55.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:54.651+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:55.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:54.652+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:55.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:55.225+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:55.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:55.225+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:55.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:55.225+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:55.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:55.226+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:55.946 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:20:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:20:55.822+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 30775 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:20:55.946 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:55.643+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:55.946 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:55.643+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:55.946 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:55.643+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:55.947 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:55.643+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:55.947 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:55.878+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:55.947 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:55.878+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:55.947 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:55.878+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:55.947 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:55.879+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:56.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:56.205+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:56.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:56.205+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:56.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:56.205+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:56.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:56.206+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:57.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:56.833+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:57.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:56.833+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:57.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:56.833+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:57.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:56.833+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:57.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:56.640+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:57.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:56.640+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:57.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:56.640+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:57.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:56.640+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:57.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:57.208+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:57.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:57.208+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:57.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:57.208+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:57.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:57.209+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:58.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:57.607+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:58.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:57.607+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:58.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:57.607+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:58.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:57.608+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:58.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:57.826+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:58.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:57.826+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:58.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:57.826+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:58.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:57.826+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:58.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:58.177+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:58.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:58.177+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:58.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:58.177+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:58.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:58.178+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:59.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:58.648+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:59.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:58.648+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:59.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:58.648+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:20:59.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:58.649+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:59.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:58.777+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:59.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:58.777+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:59.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:58.777+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:20:59.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:58.778+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:20:59.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:59.202+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:59.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:59.202+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:59.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:59.202+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:20:59.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:20:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:20:59.203+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:00.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:59.800+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:00.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:59.800+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:00.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:59.800+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:00.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:20:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:20:59.801+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:00.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:59.675+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:00.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:59.675+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:00.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:59.675+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:00.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:20:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:20:59.676+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:00.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:00.239+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:00.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:00.239+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:00.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:00.239+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:00.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:00.240+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:00.949 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:21:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:21:00.826+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 30863 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:21:00.949 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:00.688+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:00.949 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:00.688+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:00.949 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:00.688+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:00.950 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:00.689+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:00.950 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:00.813+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:00.950 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:00.813+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:00.950 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:00.813+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:00.950 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:00.814+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:01.641 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:01.287+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:01.641 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:01.287+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:01.642 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:01.287+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:01.642 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:01.288+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:02.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:01.847+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:02.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:01.847+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:02.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:01.847+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:02.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:01.848+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:02.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:01.641+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:02.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:01.641+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:02.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:01.641+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:02.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:01.641+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:02.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:02.248+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:02.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:02.248+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:02.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:02.248+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:02.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:02.249+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:03.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:02.594+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:03.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:02.594+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:03.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:02.594+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:03.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:02.595+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:03.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:02.801+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:03.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:02.801+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:03.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:02.801+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:03.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:02.802+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:03.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:03.263+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:03.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:03.263+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:03.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:03.263+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:03.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:03.265+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:04.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:03.635+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:04.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:03.635+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:04.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:03.635+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:04.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:03.635+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:04.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:03.772+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:04.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:03.772+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:04.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:03.772+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:04.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:03.773+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:04.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:04.276+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:04.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:04.276+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:04.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:04.276+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:04.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:04.277+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:05.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:04.671+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:05.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:04.671+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:05.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:04.671+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:05.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:04.671+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:05.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:04.790+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:05.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:04.790+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:05.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:04.790+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:05.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:04.791+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:05.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:05.279+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:05.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:05.279+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:05.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:05.279+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:05.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:05.280+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:05.953 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:21:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:21:05.829+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 30954 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:21:05.954 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:05.651+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:05.954 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:05.651+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:05.954 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:05.651+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:05.954 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:05.652+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:05.954 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:05.776+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:05.954 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:05.776+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:05.955 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:05.776+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:05.955 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:05.777+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:06.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:06.267+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:06.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:06.267+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:06.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:06.267+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:06.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:06.268+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:07.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:06.761+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:07.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:06.761+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:07.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:06.761+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:07.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:06.762+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:07.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:06.696+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:07.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:06.696+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:07.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:06.696+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:07.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:06.697+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:07.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:07.272+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:07.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:07.272+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:07.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:07.272+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:07.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:07.273+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:08.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:07.726+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:08.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:07.726+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:08.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:07.726+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:08.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:07.727+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:08.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:07.651+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:08.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:07.651+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:08.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:07.651+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:08.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:07.651+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:08.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:08.292+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:08.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:08.292+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:08.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:08.292+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:08.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:08.293+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:09.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:08.615+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:09.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:08.615+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:09.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:08.615+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:09.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:08.616+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:09.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:08.708+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:09.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:08.708+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:09.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:08.708+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:09.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:08.709+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:09.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:09.273+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:09.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:09.273+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:09.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:09.273+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:09.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:09.274+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:09.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:09.566+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:09.855 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:09.566+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:09.855 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:09.566+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:09.855 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:09.567+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:09.855 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:09.736+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:09.855 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:09.736+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:09.855 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:09.736+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:09.856 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:09.736+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:10.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:10.232+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:10.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:10.232+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:10.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:10.232+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:10.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:10.232+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:10.856 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:21:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:21:10.834+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 31042 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:21:10.857 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:10.603+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:10.857 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:10.603+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:10.857 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:10.603+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:10.857 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:10.604+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:10.857 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:10.696+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:10.857 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:10.696+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:10.857 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:10.696+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:10.858 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:10.697+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:11.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:11.258+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:11.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:11.258+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:11.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:11.258+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:11.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:11.259+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:12.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:11.573+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:12.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:11.573+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:12.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:11.573+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:12.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:11.574+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:12.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:11.737+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:12.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:11.737+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:12.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:11.737+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:12.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:11.738+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:12.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:12.228+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:12.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:12.228+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:12.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:12.228+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:12.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:12.229+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:13.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:12.538+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:13.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:12.538+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:13.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:12.538+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:13.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:12.539+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:13.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:12.692+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:13.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:12.692+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:13.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:12.692+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:13.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:12.693+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:13.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:13.262+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:13.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:13.262+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:13.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:13.262+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:13.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:13.263+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:13.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:13.506+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:13.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:13.506+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:13.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:13.506+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:13.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:13.507+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:14.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:13.717+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:14.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:13.717+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:14.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:13.717+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:14.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:13.718+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:14.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:14.475+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:14.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:14.475+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:14.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:14.475+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:14.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:14.475+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:14.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:14.257+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:14.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:14.257+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:14.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:14.257+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:14.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:14.257+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:15.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:14.679+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:15.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:14.679+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:15.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:14.679+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:15.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:14.679+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:15.344 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:15.254+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:15.344 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:15.254+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:15.345 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:15.254+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:15.345 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:15.254+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:15.716 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:15.428+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:15.716 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:15.429+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:15.716 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:15.429+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:15.716 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:15.429+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:16.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:21:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:21:15.838+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 31131 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:21:16.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:15.716+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:16.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:15.716+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:16.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:15.716+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:16.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:15.716+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:16.473 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:16.209+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:16.473 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:16.209+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:16.474 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:16.209+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:16.474 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:16.209+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:16.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:16.474+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:16.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:16.474+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:16.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:16.474+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:16.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:16.474+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:16.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:16.703+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:16.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:16.703+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:16.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:16.703+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:16.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:16.703+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:17.515 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:17.242+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:17.516 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:17.242+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:17.516 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:17.242+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:17.516 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:17.242+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:17.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:17.739+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:17.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:17.739+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:17.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:17.739+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:17.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:17.740+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:17.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:17.515+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:17.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:17.515+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:17.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:17.515+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:17.785 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:17.515+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:18.532 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:18.260+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:18.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:18.260+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:18.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:18.260+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:18.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:18.261+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:18.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:18.497+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:18.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:18.497+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:18.534 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:18.497+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:18.535 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:18.498+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:19.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:18.780+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:19.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:18.780+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:19.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:18.780+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:19.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:18.781+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:19.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:19.308+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:19.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:19.308+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:19.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:19.308+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:19.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:19.309+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:19.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:19.495+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:19.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:19.495+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:19.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:19.495+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:19.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:19.495+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:20.208 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:19.824+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:20.208 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:19.824+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:20.208 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:19.824+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:20.208 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:19.825+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:20.528 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:20.260+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:20.528 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:20.260+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:20.529 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:20.260+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:20.529 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:20.261+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:20.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:20.529+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:20.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:20.529+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:20.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:20.529+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:20.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:20.529+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:21.218 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:21:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:21:20.842+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 31220 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:21:21.218 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:20.848+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:21.218 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:20.848+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:21.219 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:20.848+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:21.219 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:20.850+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:21.488 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:21.218+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:21.489 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:21.218+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:21.489 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:21.218+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:21.489 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:21.220+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:21.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:21.524+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:21.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:21.524+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:21.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:21.524+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:21.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:21.524+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:22.196 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:21.893+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:22.196 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:21.893+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:22.196 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:21.893+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:22.196 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:21.894+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:22.493 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:22.197+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:22.493 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:22.197+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:22.494 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:22.197+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:22.494 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:22.197+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:22.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:22.494+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:22.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:22.494+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:22.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:22.494+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:22.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:22.495+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:23.242 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:22.920+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:23.243 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:22.920+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:23.243 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:22.920+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:23.243 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:22.921+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:23.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:23.490+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:23.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:23.490+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:23.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:23.490+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:23.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:23.490+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:23.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:23.243+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:23.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:23.243+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:23.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:23.243+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:23.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:23.244+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:24.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:23.872+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:24.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:23.872+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:24.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:23.872+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:24.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:23.872+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:24.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:24.456+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:24.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:24.456+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:24.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:24.456+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:24.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:24.457+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:24.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:24.209+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:24.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:24.209+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:24.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:24.209+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:24.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:24.210+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:25.173 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:24.920+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:25.173 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:24.920+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:25.174 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:24.920+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:25.174 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:24.921+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:25.431 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:25.174+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:25.431 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:25.174+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:25.431 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:25.174+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:25.432 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:25.175+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:25.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:25.432+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:25.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:25.432+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:25.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:25.432+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:25.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:25.432+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:26.199 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:21:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:21:25.846+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 31310 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:21:26.199 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:25.895+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:26.199 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:25.895+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:26.199 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:25.895+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:26.199 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:25.896+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:26.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:26.433+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:26.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:26.433+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:26.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:26.433+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:26.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:26.433+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:26.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:26.199+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:26.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:26.199+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:26.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:26.199+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:26.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:26.201+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:27.221 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:26.872+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:27.222 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:26.872+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:27.222 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:26.872+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:27.222 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:26.873+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:27.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:27.468+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:27.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:27.468+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:27.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:27.468+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:27.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:27.469+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:27.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:27.222+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:27.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:27.222+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:27.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:27.222+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:27.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:27.223+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:28.257 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:27.876+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:28.257 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:27.876+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:28.257 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:27.876+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:28.258 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:27.876+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:28.443+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:28.443+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:28.443+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:28.443+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:28.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:28.258+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:28.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:28.258+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:28.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:28.258+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:28.533 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:28.259+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:29.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:28.925+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:29.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:28.925+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:29.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:28.925+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:29.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:28.925+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:29.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:29.437+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:29.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:29.437+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:29.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:29.437+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:29.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:29.438+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:29.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:29.292+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:29.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:29.292+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:29.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:29.292+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:29.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:29.293+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:30.209 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:29.895+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:30.209 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:29.895+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:30.209 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:29.895+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:30.210 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:29.895+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:30.481 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:30.329+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:30.481 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:30.329+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:30.481 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:30.329+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:30.481 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:30.330+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:30.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:30.480+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:30.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:30.480+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:30.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:30.480+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:30.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:30.481+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:31.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:21:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:21:30.849+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 31400 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:21:31.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:30.934+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:31.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:30.935+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:31.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:30.935+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:31.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:30.935+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:31.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:31.471+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:31.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:31.471+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:31.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:31.471+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:31.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:31.472+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:31.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:31.320+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:31.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:31.320+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:31.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:31.320+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:31.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:31.321+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:32.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:31.935+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:32.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:31.935+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:32.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:31.935+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:32.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:31.936+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:32.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:32.487+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:32.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:32.487+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:32.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:32.487+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:32.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:32.489+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:32.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:32.286+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:32.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:32.286+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:32.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:32.286+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:32.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:32.287+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:33.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:32.975+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:33.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:32.975+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:33.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:32.975+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:33.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:32.976+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:33.639 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:33.443+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:33.640 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:33.443+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:33.640 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:33.443+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:33.640 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:33.444+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:33.640 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:33.325+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:33.640 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:33.325+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:33.640 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:33.325+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:33.640 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:33.326+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:34.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:33.969+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:34.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:33.969+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:34.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:33.969+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:34.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:33.971+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:34.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:34.418+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:34.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:34.418+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:34.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:34.418+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:34.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:34.419+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:34.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:34.349+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:34.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:34.350+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:34.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:34.350+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:34.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:34.350+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:35.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:34.983+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:35.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:34.983+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:35.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:34.983+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:35.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:34.984+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:35.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:35.452+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:35.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:35.452+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:35.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:35.452+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:35.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:35.453+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:35.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:35.397+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:35.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:35.397+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:35.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:35.397+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:35.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:35.398+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:36.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:21:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:21:35.853+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 31485 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:21:36.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:36.001+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:36.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:36.001+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:36.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:36.001+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:36.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:36.002+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:36.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:36.441+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:36.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:36.441+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:36.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:36.441+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:36.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:36.442+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:36.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:36.416+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:36.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:36.416+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:36.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:36.416+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:36.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:36.416+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:37.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:37.029+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:37.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:37.029+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:37.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:37.029+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:37.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:37.029+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:37.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:37.437+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:37.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:37.437+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:37.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:37.437+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:37.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:37.438+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:37.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:37.419+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:37.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:37.419+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:37.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:37.419+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:37.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:37.420+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:38.400 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:38.066+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:38.400 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:38.066+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:38.400 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:38.066+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:38.400 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:38.066+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:38.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:38.399+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:38.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:38.399+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:38.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:38.399+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:38.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:38.400+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:38.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:38.473+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:38.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:38.473+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:38.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:38.473+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:38.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:38.474+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:39.366 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:39.070+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:39.367 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:39.070+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:39.367 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:39.070+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:39.367 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:39.071+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:39.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:39.445+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:39.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:39.445+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:39.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:39.445+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:39.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:39.446+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:39.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:39.366+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:39.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:39.366+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:39.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:39.366+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:39.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:39.366+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:40.210 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:40.052+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:40.210 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:40.052+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:40.210 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:40.052+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:40.211 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:40.053+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:40.479 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:40.389+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:40.479 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:40.389+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:40.479 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:40.389+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:40.479 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:40.390+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:40.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:40.478+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:40.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:40.478+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:40.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:40.478+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:40.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:40.479+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:41.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:21:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:21:40.857+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 31574 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:21:41.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:41.027+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:41.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:41.027+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:41.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:41.027+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:41.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:41.028+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:41.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:41.515+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:41.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:41.515+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:41.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:41.515+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:41.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:41.516+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:41.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:41.342+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:41.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:41.343+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:41.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:41.343+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:41.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:41.344+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:42.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:42.020+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:42.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:42.020+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:42.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:42.020+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:42.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:42.021+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:42.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:42.539+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:42.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:42.539+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:42.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:42.539+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:42.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:42.540+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:42.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:42.391+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:42.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:42.391+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:42.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:42.391+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:42.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:42.392+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:43.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:43.029+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:43.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:43.029+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:43.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:43.029+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:43.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:43.029+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:43.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:43.574+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:43.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:43.574+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:43.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:43.574+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:43.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:43.574+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:43.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:43.413+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:43.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:43.413+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:43.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:43.413+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:43.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:43.414+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:44.458 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:44.075+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:44.458 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:44.075+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:44.459 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:44.075+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:44.459 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:44.076+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:44.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:44.458+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:44.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:44.458+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:44.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:44.458+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:44.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:44.458+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:44.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:44.600+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:44.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:44.600+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:44.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:44.600+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:44.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:44.600+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:45.424 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:45.112+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:45.425 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:45.112+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:45.425 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:45.112+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:45.425 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:45.113+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:45.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:45.424+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:45.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:45.424+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:45.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:45.424+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:45.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:45.425+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:45.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:45.592+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:45.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:45.592+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:45.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:45.592+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:45.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:45.592+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:46.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:21:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:21:45.860+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 31664 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:21:46.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:46.099+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:46.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:46.099+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:46.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:46.099+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:46.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:46.100+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:46.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:46.470+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:46.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:46.470+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:46.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:46.470+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:46.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:46.471+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:46.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:46.545+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:46.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:46.545+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:46.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:46.545+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:46.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:46.545+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:47.471 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:47.093+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:47.471 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:47.093+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:47.471 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:47.093+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:47.472 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:47.094+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:47.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:47.591+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:47.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:47.591+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:47.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:47.591+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:47.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:47.592+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:47.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:47.470+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:47.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:47.470+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:47.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:47.470+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:47.784 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:47.471+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:48.469 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:48.076+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:48.470 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:48.076+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:48.470 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:48.076+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:48.470 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:48.077+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:48.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:48.562+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:48.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:48.562+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:48.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:48.562+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:48.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:48.563+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:48.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:48.468+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:48.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:48.468+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:48.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:48.468+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:48.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:48.470+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:49.502 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:49.047+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:49.502 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:49.047+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:49.502 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:49.047+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:49.502 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:49.048+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:49.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:49.513+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:49.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:49.513+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:49.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:49.513+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:49.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:49.514+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:49.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:49.501+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:49.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:49.501+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:49.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:49.501+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:49.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:49.502+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:50.211 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:50.017+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:50.212 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:50.017+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:50.212 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:50.017+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:50.212 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:50.018+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:50.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:50.498+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:50.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:50.498+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:50.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:50.498+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:50.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:50.498+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:50.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:50.467+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:50.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:50.467+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:50.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:50.467+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:50.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:50.468+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:51.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:21:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:21:50.865+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 31754 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:21:51.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:50.987+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:51.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:50.988+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:51.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:50.988+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:51.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:50.989+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:51.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:51.472+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:51.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:51.472+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:51.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:51.472+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:51.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:51.472+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:51.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:51.457+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:51.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:51.457+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:51.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:51.457+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:51.784 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:51.457+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:52.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:52.004+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:52.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:52.004+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:52.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:52.004+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:52.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:52.005+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:52.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:52.441+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:52.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:52.441+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:52.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:52.441+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:52.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:52.442+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:52.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:52.491+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:52.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:52.491+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:52.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:52.491+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:52.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:52.492+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:53.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:53.012+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:53.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:53.012+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:53.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:53.012+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:53.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:53.013+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:53.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:53.396+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:53.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:53.396+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:53.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:53.396+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:53.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:53.397+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:53.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:53.542+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:53.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:53.542+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:53.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:53.542+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:53.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:53.543+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:54.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:54.001+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:54.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:54.001+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:54.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:54.001+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:54.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:54.002+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:54.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:54.414+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:54.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:54.414+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:54.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:54.414+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:54.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:54.415+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:54.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:54.538+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:54.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:54.538+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:54.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:54.538+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:54.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:54.538+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:55.366 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:55.050+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:55.367 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:55.050+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:55.367 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:55.050+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:55.367 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:55.050+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:55.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:55.512+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:55.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:55.512+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:55.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:55.512+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:55.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:55.514+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:55.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:55.366+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:55.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:55.366+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:55.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:55.366+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:55.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:55.366+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:56.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:21:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:21:55.868+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 31838 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:21:56.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:56.090+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:56.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:56.090+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:56.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:56.090+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:56.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:56.091+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:56.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:56.478+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:56.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:56.478+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:56.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:56.478+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:56.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:56.479+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:56.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:56.342+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:56.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:56.342+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:56.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:56.342+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:56.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:56.342+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:57.380 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:57.096+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:57.380 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:57.096+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:57.381 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:57.096+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:57.381 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:57.097+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:57.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:57.508+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:57.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:57.508+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:57.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:57.508+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:57.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:57.509+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:57.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:57.380+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:57.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:57.380+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:57.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:57.380+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:57.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:57.380+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:58.365 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:58.092+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:58.366 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:58.092+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:58.367 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:58.092+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:58.367 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:58.093+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:58.686 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:58.466+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:58.686 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:58.466+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:58.686 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:58.466+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:58.686 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:58.467+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:58.686 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:58.365+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:58.687 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:58.365+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:58.687 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:58.365+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:58.687 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:58.366+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:59.493 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:59.124+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:59.494 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:59.124+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:59.494 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:59.124+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:21:59.494 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:21:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:21:59.125+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:59.494 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:59.320+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:59.494 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:59.320+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:59.495 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:59.320+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:21:59.495 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:21:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:21:59.321+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:21:59.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:59.493+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:59.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:59.493+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:59.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:59.493+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:21:59.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:21:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:21:59.494+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:00.211 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:00.079+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:00.212 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:00.079+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:00.212 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:00.079+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:00.212 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:00.081+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:00.482 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:00.298+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:00.482 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:00.298+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:00.482 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:00.298+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:00.482 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:00.299+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:00.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:00.481+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:00.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:00.481+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:00.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:00.481+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:00.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:00.482+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:01.261 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:22:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:22:00.872+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 31925 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:22:01.262 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:01.103+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:01.262 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:01.103+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:01.262 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:01.103+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:01.262 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:01.104+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:01.520 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:01.261+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:01.520 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:01.261+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:01.520 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:01.261+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:01.521 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:01.262+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:01.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:01.520+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:01.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:01.520+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:01.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:01.520+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:01.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:01.520+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:02.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:02.116+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:02.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:02.116+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:02.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:02.116+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:02.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:02.117+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:02.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:02.214+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:02.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:02.214+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:02.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:02.214+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:02.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:02.215+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:03.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:02.566+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:03.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:02.566+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:03.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:02.566+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:03.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:02.567+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:03.523 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:03.249+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:03.523 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:03.249+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:03.523 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:03.249+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:03.523 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:03.249+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:03.523 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:03.119+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:03.524 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:03.119+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:03.524 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:03.119+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:03.524 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:03.119+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:03.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:03.522+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:03.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:03.522+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:03.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:03.522+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:03.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:03.523+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:04.496 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:04.226+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:04.497 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:04.226+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:04.497 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:04.226+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:04.497 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:04.226+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:04.497 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:04.103+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:04.497 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:04.103+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:04.497 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:04.103+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:04.498 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:04.104+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:04.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:04.573+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:04.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:04.573+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:04.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:04.573+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:04.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:04.573+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:05.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:05.198+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:05.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:05.198+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:05.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:05.198+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:05.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:05.199+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:05.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:05.127+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:05.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:05.127+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:05.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:05.127+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:05.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:05.128+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:05.877 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:05.537+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:05.877 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:05.537+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:05.877 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:05.537+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:05.878 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:05.538+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:06.195 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:22:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:22:05.876+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 32008 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:22:06.195 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:06.079+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:06.195 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:06.079+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:06.196 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:06.080+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:06.196 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:06.080+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:06.531 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:06.194+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:06.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:06.194+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:06.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:06.194+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:06.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:06.195+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:06.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:06.531+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:06.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:06.531+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:06.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:06.531+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:06.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:06.532+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:07.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:07.161+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:07.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:07.161+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:07.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:07.161+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:07.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:07.161+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:07.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:07.047+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:07.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:07.047+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:07.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:07.047+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:07.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:07.048+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:08.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:07.539+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:08.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:07.539+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:08.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:07.539+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:08.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:07.540+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:08.499 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:08.148+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:08.499 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:08.148+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:08.500 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:08.148+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:08.500 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:08.149+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:08.500 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:08.055+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:08.500 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:08.055+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:08.500 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:08.055+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:08.500 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:08.056+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:08.772 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:08.499+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:08.772 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:08.499+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:08.773 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:08.499+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:08.773 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:08.499+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:09.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:09.019+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:09.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:09.019+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:09.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:09.019+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:09.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:09.020+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:09.510 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:09.138+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:09.510 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:09.138+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:09.510 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:09.138+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:09.510 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:09.139+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:09.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:09.509+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:09.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:09.509+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:09.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:09.509+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:09.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:09.510+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:10.187 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:10.015+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:10.187 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:10.015+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:10.187 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:10.015+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:10.188 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:10.015+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:10.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:10.187+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:10.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:10.187+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:10.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:10.187+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:10.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:10.188+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:10.881 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:10.552+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:10.881 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:10.552+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:10.881 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:10.552+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:10.881 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:10.553+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:11.181 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:22:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:22:10.880+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 32087 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:22:11.181 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:11.018+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:11.181 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:11.018+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:11.181 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:11.018+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:11.181 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:11.019+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:11.518 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:11.180+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:11.519 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:11.180+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:11.519 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:11.180+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:11.519 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:11.182+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:11.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:11.518+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:11.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:11.518+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:11.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:11.518+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:11.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:11.519+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:12.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:12.166+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:12.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:12.166+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:12.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:12.166+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:12.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:12.167+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:12.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:12.000+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:12.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:12.000+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:12.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:12.000+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:12.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:12.000+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:13.029 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:12.568+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:13.030 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:12.568+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:13.030 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:12.568+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:13.030 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:12.568+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:13.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:13.150+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:13.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:13.150+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:13.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:13.150+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:13.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:13.150+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:13.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:13.029+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:13.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:13.029+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:13.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:13.029+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:13.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:13.030+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:14.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:13.573+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:14.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:13.573+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:14.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:13.573+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:14.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:13.574+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:14.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:14.079+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:14.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:14.079+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:14.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:14.079+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:14.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:14.080+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:14.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:14.106+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:14.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:14.106+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:14.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:14.106+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:14.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:14.107+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:15.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:14.608+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:15.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:14.608+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:15.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:14.608+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:15.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:14.609+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:15.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:15.137+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:15.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:15.137+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:15.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:15.137+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:15.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:15.138+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:15.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:15.040+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:15.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:15.040+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:15.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:15.040+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:15.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:15.041+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:15.885 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:15.562+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:15.885 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:15.562+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:15.885 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:15.562+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:15.886 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:15.563+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:16.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:22:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:22:15.884+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 32165 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:22:16.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:16.019+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:16.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:16.019+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:16.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:16.019+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:16.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:16.020+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:16.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:16.102+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:16.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:16.102+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:16.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:16.102+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:16.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:16.103+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:16.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:16.522+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:16.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:16.522+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:16.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:16.522+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:16.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:16.523+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:17.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:17.056+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:17.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:17.056+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:17.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:17.056+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:17.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:17.057+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:17.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:17.030+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:17.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:17.030+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:17.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:17.030+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:17.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:17.031+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:17.984 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:17.543+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:17.985 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:17.543+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:17.985 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:17.543+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:17.985 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:17.543+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:18.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:17.983+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:18.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:17.983+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:18.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:17.983+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:18.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:17.984+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:18.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:18.079+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:18.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:18.079+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:18.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:18.079+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:18.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:18.080+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:18.850 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:18.573+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:18.850 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:18.573+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:18.851 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:18.573+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:18.851 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:18.573+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:19.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:19.072+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:19.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:19.072+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:19.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:19.072+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:19.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:19.073+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:19.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:19.006+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:19.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:19.006+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:19.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:19.006+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:19.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:19.007+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:19.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:19.525+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:19.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:19.525+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:19.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:19.525+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:19.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:19.526+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:20.213 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:20.084+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:20.213 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:20.084+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:20.214 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:20.084+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:20.214 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:20.085+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:20.214 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:20.022+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:20.214 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:20.022+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:20.214 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:20.022+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:20.214 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:20.023+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:20.889 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:20.542+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:20.889 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:20.542+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:20.889 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:20.542+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:20.889 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:20.542+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:21.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:22:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:22:20.888+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 32249 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:22:21.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:21.115+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:21.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:21.115+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:21.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:21.115+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:21.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:21.116+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:21.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:21.044+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:21.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:21.044+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:21.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:21.044+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:21.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:21.045+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:22.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:21.586+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:22.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:21.586+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:22.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:21.586+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:22.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:21.586+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:22.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:22.115+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:22.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:22.115+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:22.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:22.115+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:22.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:22.116+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:22.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:22.040+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:22.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:22.040+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:22.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:22.040+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:22.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:22.040+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:22.928 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:22.578+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:22.929 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:22.578+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:22.929 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:22.578+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:22.929 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:22.579+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:23.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:23.072+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:23.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:23.072+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:23.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:23.072+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:23.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:23.073+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:23.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:23.130+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:23.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:23.130+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:23.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:23.130+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:23.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:23.130+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:23.889 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:23.577+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:23.889 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:23.577+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:23.889 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:23.577+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:23.890 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:23.578+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:24.156 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:24.095+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:24.157 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:24.095+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:24.157 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:24.095+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:24.157 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:24.096+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:24.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:24.155+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:24.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:24.155+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:24.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:24.155+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:24.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:24.156+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:25.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:24.548+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:25.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:24.548+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:25.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:24.548+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:25.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:24.548+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:25.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:25.090+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:25.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:25.090+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:25.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:25.090+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:25.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:25.092+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:25.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:25.135+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:25.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:25.135+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:25.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:25.135+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:25.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:25.136+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:25.892 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:25.568+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:25.893 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:25.568+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:25.893 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:25.568+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:25.893 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:25.569+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:26.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:22:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:22:25.892+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 32332 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:22:26.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:26.110+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:26.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:26.110+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:26.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:26.110+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:26.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:26.111+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:26.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:26.126+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:26.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:26.126+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:26.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:26.126+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:26.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:26.126+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:27.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:26.534+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:27.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:26.534+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:27.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:26.534+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:27.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:26.535+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:27.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:27.092+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:27.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:27.092+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:27.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:27.092+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:27.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:27.094+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:27.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:27.080+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:27.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:27.080+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:27.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:27.080+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:27.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:27.081+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:28.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:27.566+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:28.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:27.566+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:28.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:27.566+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:28.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:27.567+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:28.084+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:28.084+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:28.084+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:28.085+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:28.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:28.062+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:28.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:28.062+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:28.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:28.062+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:28.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:28.063+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:28.930 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:28.595+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:28.931 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:28.595+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:28.931 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:28.595+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:28.931 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:28.596+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:29.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:29.109+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:29.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:29.109+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:29.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:29.109+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:29.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:29.110+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:29.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:29.105+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:29.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:29.105+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:29.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:29.105+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:29.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:29.106+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:30.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:29.632+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:30.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:29.632+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:30.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:29.632+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:30.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:29.633+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:30.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:30.149+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:30.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:30.149+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:30.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:30.149+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:30.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:30.149+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:30.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:30.141+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:30.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:30.141+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:30.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:30.141+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:30.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:30.141+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:30.897 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:30.590+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:30.897 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:30.590+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:30.897 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:30.590+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:30.898 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:30.591+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:31.171 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:22:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:22:30.896+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 32422 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:22:31.171 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:31.105+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:31.171 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:31.105+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:31.171 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:31.105+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:31.171 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:31.105+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:31.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:31.170+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:31.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:31.170+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:31.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:31.170+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:31.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:31.171+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:32.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:31.627+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:32.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:31.627+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:32.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:31.627+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:32.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:31.628+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:32.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:32.172+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:32.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:32.172+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:32.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:32.172+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:32.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:32.173+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:32.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:32.152+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:32.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:32.152+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:32.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:32.152+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:32.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:32.153+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:33.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:32.663+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:33.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:32.663+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:33.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:32.663+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:33.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:32.664+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:33.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:33.137+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:33.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:33.137+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:33.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:33.137+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:33.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:33.138+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:33.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:33.147+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:33.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:33.147+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:33.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:33.147+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:33.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:33.148+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:33.971 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:33.696+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:33.971 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:33.696+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:33.971 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:33.696+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:33.971 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:33.697+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:34.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:34.105+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:34.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:34.105+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:34.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:34.105+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:34.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:34.106+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:34.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:34.141+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:34.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:34.141+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:34.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:34.141+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:34.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:34.142+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:35.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:34.706+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:35.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:34.706+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:35.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:34.706+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:35.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:34.707+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:35.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:35.145+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:35.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:35.145+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:35.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:35.145+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:35.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:35.145+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:35.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:35.164+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:35.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:35.164+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:35.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:35.164+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:35.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:35.164+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:36.023 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:35.725+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:36.023 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:35.725+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:36.023 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:35.725+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:36.024 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:35.726+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:36.024 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:22:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:22:35.900+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 32509 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:22:36.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:36.185+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:36.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:36.185+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:36.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:36.185+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:36.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:36.186+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:36.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:36.135+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:36.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:36.135+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:36.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:36.135+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:36.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:36.136+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:37.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:36.776+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:37.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:36.776+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:37.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:36.776+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:37.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:36.776+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:37.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:37.184+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:37.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:37.184+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:37.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:37.184+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:37.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:37.184+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:37.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:37.099+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:37.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:37.099+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:37.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:37.099+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:37.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:37.100+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:38.178 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:38.068+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:38.179 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:38.068+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:38.179 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:38.068+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:38.179 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:38.069+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:38.180 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:37.820+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:38.180 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:37.820+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:38.180 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:37.820+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:38.180 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:37.820+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:38.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:38.175+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:38.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:38.175+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:38.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:38.175+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:38.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:38.176+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:39.201 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:38.783+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:39.202 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:38.783+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:39.202 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:38.783+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:39.202 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:38.783+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:39.202 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:39.027+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:39.202 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:39.027+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:39.203 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:39.027+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:39.203 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:39.028+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:39.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:39.202+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:39.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:39.202+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:39.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:39.202+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:39.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:39.202+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:40.181 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:39.782+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:40.181 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:39.782+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:40.181 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:39.782+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:40.181 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:39.784+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:40.181 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:40.014+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:40.182 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:40.014+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:40.182 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:40.014+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:40.182 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:40.014+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:40.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:40.181+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:40.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:40.181+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:40.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:40.181+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:40.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:40.182+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:41.199 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:22:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:22:40.905+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 32599 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:22:41.199 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:40.811+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:41.199 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:40.812+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:41.199 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:40.812+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:41.199 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:40.812+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:41.200 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:41.015+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:41.200 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:41.015+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:41.200 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:41.015+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:41.200 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:41.016+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:41.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:41.199+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:41.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:41.199+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:41.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:41.199+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:41.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:41.200+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:42.225 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:41.827+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:42.225 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:41.827+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:42.226 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:41.827+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:42.226 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:41.828+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:42.226 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:42.018+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:42.226 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:42.018+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:42.226 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:42.018+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:42.227 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:42.018+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:42.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:42.225+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:42.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:42.225+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:42.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:42.225+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:42.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:42.226+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:43.207 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:42.839+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:43.208 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:42.839+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:43.208 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:42.839+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:43.208 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:42.840+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:43.208 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:42.999+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:43.208 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:42.999+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:43.209 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:42.999+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:43.209 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:43.000+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:43.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:43.208+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:43.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:43.208+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:43.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:43.208+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:43.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:43.209+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:44.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:43.805+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:44.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:43.805+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:44.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:43.805+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:44.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:43.805+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:44.433 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:44.226+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:44.433 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:44.226+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:44.433 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:44.226+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:44.433 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:44.228+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:44.433 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:44.034+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:44.434 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:44.034+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:44.434 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:44.034+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:44.434 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:44.035+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:45.182 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:44.824+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:45.182 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:44.824+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:45.182 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:44.824+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:45.182 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:44.826+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:45.183 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:45.019+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:45.183 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:45.019+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:45.183 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:45.019+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:45.183 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:45.020+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:45.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:45.182+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:45.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:45.182+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:45.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:45.182+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:45.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:45.183+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:46.191 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:22:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:22:45.909+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 32688 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:22:46.191 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:45.832+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:46.191 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:45.832+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:46.191 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:45.832+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:46.191 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:45.833+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:46.192 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:46.039+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:46.192 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:46.039+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:46.192 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:46.039+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:46.192 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:46.039+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:46.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:46.191+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:46.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:46.191+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:46.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:46.191+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:46.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:46.192+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:47.047 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:46.792+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:47.047 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:46.792+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:47.047 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:46.792+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:47.047 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:46.794+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:47.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:47.047+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:47.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:47.047+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:47.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:47.047+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:47.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:47.048+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:47.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:47.231+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:47.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:47.231+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:47.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:47.231+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:47.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:47.231+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:48.197 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:47.827+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:48.198 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:47.827+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:48.198 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:47.827+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:48.198 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:47.828+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:48.199 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:48.055+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:48.199 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:48.055+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:48.199 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:48.055+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:48.199 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:48.056+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:48.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:48.198+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:48.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:48.198+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:48.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:48.198+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:48.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:48.199+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:49.091 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:48.823+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:49.091 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:48.823+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:49.091 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:48.823+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:49.091 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:48.824+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:49.092 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:49.046+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:49.092 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:49.046+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:49.092 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:49.046+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:49.092 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:49.047+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:49.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:49.150+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:49.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:49.150+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:49.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:49.150+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:49.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:49.150+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:50.191 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:50.000+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:50.191 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:50.000+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:50.191 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:50.000+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:50.191 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:50.002+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:50.191 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:49.844+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:50.192 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:49.844+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:50.192 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:49.844+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:50.192 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:49.845+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:50.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:50.191+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:50.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:50.191+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:50.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:50.191+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:50.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:50.192+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:51.233 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:22:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:22:50.913+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 32778 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:22:51.233 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:50.978+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:51.233 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:50.978+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:51.233 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:50.978+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:51.233 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:50.979+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:51.234 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:50.842+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:51.234 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:50.842+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:51.234 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:50.842+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:51.234 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:50.844+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:51.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:51.233+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:51.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:51.233+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:51.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:51.233+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:51.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:51.235+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:52.276 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:51.878+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:52.277 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:51.878+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:52.277 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:51.879+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:52.277 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:51.879+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:52.277 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:51.963+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:52.278 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:51.963+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:52.278 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:51.963+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:52.278 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:51.964+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:52.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:52.275+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:52.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:52.275+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:52.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:52.275+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:52.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:52.277+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:53.265 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:52.879+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:53.265 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:52.879+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:53.265 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:52.879+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:53.265 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:52.880+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:53.265 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:52.992+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:53.265 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:52.992+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:53.266 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:52.992+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:53.266 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:52.993+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:53.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:53.264+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:53.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:53.264+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:53.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:53.264+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:53.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:53.265+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:53.980 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:53.860+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:53.981 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:53.860+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:53.981 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:53.860+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:53.981 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:53.860+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:54.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:53.980+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:54.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:53.980+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:54.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:53.980+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:54.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:53.981+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:54.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:54.314+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:54.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:54.314+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:54.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:54.314+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:54.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:54.315+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:55.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:54.845+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:55.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:54.845+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:55.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:54.845+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:55.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:54.846+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:55.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:54.951+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:55.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:54.951+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:55.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:54.951+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:55.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:54.952+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:55.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:55.349+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:55.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:55.349+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:55.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:55.349+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:55.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:55.350+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:56.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:22:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:22:55.916+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 32866 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:22:56.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:55.888+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:56.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:55.889+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:56.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:55.889+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:56.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:55.889+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:56.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:55.944+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:56.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:55.944+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:56.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:55.944+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:56.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:55.945+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:56.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:56.309+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:56.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:56.309+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:56.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:56.309+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:56.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:56.310+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:57.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:56.898+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:57.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:56.898+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:57.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:56.898+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:57.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:56.899+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:57.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:56.850+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:57.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:56.850+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:57.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:56.850+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:57.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:56.851+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:57.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:57.320+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:57.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:57.320+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:57.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:57.320+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:57.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:57.321+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:58.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:57.897+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:58.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:57.897+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:58.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:57.897+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:58.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:57.898+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:58.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:57.893+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:58.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:57.893+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:58.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:57.893+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:58.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:57.894+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:58.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:58.304+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:58.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:58.304+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:58.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:58.304+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:58.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:58.305+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:59.176 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:58.905+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:59.177 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:58.905+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:59.177 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:58.905+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:22:59.177 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:58.906+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:59.177 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:58.887+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:59.177 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:58.887+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:59.178 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:58.887+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:22:59.178 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:58.888+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:22:59.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:59.319+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:59.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:59.319+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:59.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:59.319+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:22:59.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:22:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:22:59.320+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:00.216 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:59.939+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:00.216 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:59.939+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:00.216 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:59.939+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:00.216 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:22:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:22:59.940+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:00.216 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:59.885+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:00.217 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:59.885+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:00.217 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:59.885+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:00.217 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:22:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:22:59.886+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:00.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:00.276+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:00.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:00.276+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:00.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:00.276+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:00.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:00.277+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:01.265 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:23:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:23:00.920+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 32956 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:23:01.266 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:00.929+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:01.266 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:00.929+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:01.266 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:00.929+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:01.266 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:00.930+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:01.266 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:00.947+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:01.266 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:00.947+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:01.267 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:00.947+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:01.267 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:00.948+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:01.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:01.265+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:01.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:01.265+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:01.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:01.265+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:01.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:01.265+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:02.246 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:01.917+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:02.246 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:01.917+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:02.246 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:01.917+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:02.246 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:01.918+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:02.246 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:01.908+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:02.247 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:01.908+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:02.247 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:01.908+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:02.247 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:01.908+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:02.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:02.245+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:02.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:02.245+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:02.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:02.245+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:02.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:02.246+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:03.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:02.939+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:03.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:02.939+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:03.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:02.939+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:03.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:02.939+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:03.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:02.902+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:03.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:02.902+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:03.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:02.902+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:03.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:02.903+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:03.646 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:03.295+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:03.647 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:03.295+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:03.647 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:03.295+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:03.647 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:03.296+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:04.219 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:03.931+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:04.219 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:03.931+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:04.220 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:03.931+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:04.220 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:03.932+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:04.220 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:03.901+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:04.220 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:03.901+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:04.220 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:03.901+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:04.220 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:03.902+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:04.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:04.336+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:04.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:04.336+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:04.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:04.336+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:04.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:04.337+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:05.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:04.859+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:05.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:04.859+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:05.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:04.859+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:05.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:04.859+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:05.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:04.961+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:05.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:04.961+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:05.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:04.961+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:05.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:04.962+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:05.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:05.300+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:05.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:05.300+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:05.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:05.300+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:05.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:05.302+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:06.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:23:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:23:05.924+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 33045 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:23:06.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:05.908+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:06.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:05.908+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:06.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:05.908+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:06.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:05.909+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:06.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:05.957+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:06.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:05.957+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:06.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:05.957+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:06.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:05.958+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:06.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:06.290+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:06.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:06.290+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:06.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:06.290+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:06.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:06.291+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:07.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:06.923+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:07.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:06.923+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:07.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:06.923+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:07.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:06.924+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:07.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:06.922+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:07.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:06.922+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:07.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:06.922+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:07.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:06.923+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:07.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:07.290+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:07.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:07.290+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:07.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:07.290+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:07.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:07.291+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:08.255 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:07.934+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:08.255 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:07.934+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:08.255 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:07.934+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:08.255 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:07.934+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:08.256 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:07.968+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:08.256 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:07.968+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:08.256 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:07.968+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:08.256 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:07.969+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:08.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:08.254+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:08.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:08.254+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:08.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:08.254+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:08.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:08.255+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:09.217 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:08.999+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:09.218 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:08.999+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:09.218 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:08.999+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:09.218 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:09.000+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:09.218 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:08.914+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:09.218 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:08.914+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:09.218 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:08.914+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:09.219 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:08.916+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:09.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:09.217+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:09.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:09.217+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:09.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:09.217+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:09.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:09.218+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:10.172 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:10.028+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:10.172 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:10.028+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:10.173 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:10.028+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:10.173 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:10.029+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:10.173 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:09.951+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:10.173 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:09.951+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:10.173 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:09.951+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:10.173 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:09.952+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:10.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:10.172+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:10.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:10.172+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:10.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:10.172+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:10.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:10.172+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:11.201 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:23:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:23:10.928+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 33135 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:23:11.202 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:11.043+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:11.202 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:11.043+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:11.202 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:11.043+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:11.202 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:11.044+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:11.203 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:10.975+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:11.203 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:10.975+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:11.203 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:10.975+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:11.203 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:10.976+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:11.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:11.201+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:11.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:11.201+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:11.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:11.201+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:11.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:11.202+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:12.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:12.034+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:12.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:12.034+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:12.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:12.034+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:12.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:12.035+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:12.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:12.005+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:12.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:12.005+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:12.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:12.005+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:12.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:12.006+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:12.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:12.216+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:12.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:12.216+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:12.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:12.216+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:12.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:12.217+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:13.261 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:13.026+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:13.261 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:13.026+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:13.261 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:13.026+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:13.261 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:13.026+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:13.262 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:12.957+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:13.262 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:12.957+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:13.262 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:12.957+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:13.262 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:12.958+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:13.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:13.260+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:13.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:13.260+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:13.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:13.260+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:13.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:13.261+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:14.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:13.987+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:14.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:13.987+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:14.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:13.987+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:14.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:13.987+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:14.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:14.215+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:14.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:14.215+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:14.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:14.215+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:14.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:14.215+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:14.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:13.979+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:14.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:13.979+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:14.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:13.979+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:14.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:13.980+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:15.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:15.176+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:15.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:15.176+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:15.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:15.176+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:15.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:15.176+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:15.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:14.985+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:15.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:14.985+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:15.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:14.985+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:15.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:14.985+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:15.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:14.988+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:15.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:14.988+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:15.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:14.988+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:15.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:14.988+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:16.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:23:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:23:15.932+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 33223 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:23:16.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:15.974+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:16.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:15.974+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:16.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:15.974+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:16.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:15.975+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:16.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:16.174+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:16.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:16.174+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:16.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:16.174+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:16.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:16.175+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:16.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:16.031+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:16.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:16.031+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:16.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:16.031+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:16.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:16.031+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:17.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:16.971+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:17.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:16.971+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:17.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:16.971+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:17.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:16.972+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:17.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:17.050+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:17.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:17.050+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:17.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:17.050+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:17.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:17.050+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:17.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:17.220+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:17.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:17.220+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:17.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:17.220+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:17.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:17.221+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:18.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:18.005+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:18.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:18.005+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:18.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:18.005+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:18.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:18.006+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:18.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:18.010+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:18.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:18.010+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:18.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:18.010+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:18.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:18.011+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:18.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:18.248+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:18.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:18.248+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:18.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:18.248+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:18.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:18.249+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:19.262 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:18.966+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:19.263 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:18.966+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:19.263 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:18.966+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:19.263 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:18.967+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:19.263 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:18.975+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:19.263 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:18.975+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:19.263 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:18.975+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:19.264 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:18.977+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:19.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:19.261+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:19.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:19.261+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:19.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:19.261+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:19.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:19.262+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:20.217 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:19.978+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:20.217 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:19.978+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:20.218 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:19.978+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:20.218 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:19.979+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:20.218 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:19.953+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:20.218 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:19.953+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:20.218 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:19.953+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:20.218 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:19.954+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:20.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:20.218+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:20.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:20.218+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:20.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:20.218+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:20.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:20.218+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:21.254 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:23:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:23:20.936+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 33313 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:23:21.254 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:20.997+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:21.254 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:20.997+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:21.255 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:20.997+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:21.255 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:20.998+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:21.255 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:20.998+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:21.255 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:20.998+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:21.255 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:20.998+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:21.255 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:20.999+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:21.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:21.253+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:21.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:21.253+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:21.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:21.253+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:21.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:21.254+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:22.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:21.985+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:22.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:21.985+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:22.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:21.985+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:22.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:21.986+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:22.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:22.027+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:22.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:22.027+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:22.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:22.027+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:22.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:22.028+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:22.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:22.301+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:22.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:22.301+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:22.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:22.301+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:22.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:22.301+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:23.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:22.955+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:23.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:22.955+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:23.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:22.955+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:23.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:22.956+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:23.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:23.059+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:23.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:23.059+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:23.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:23.059+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:23.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:23.060+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:23.647 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:23.326+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:23.648 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:23.326+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:23.648 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:23.326+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:23.648 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:23.327+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:24.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:24.025+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:24.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:24.025+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:24.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:24.025+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:24.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:24.026+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:24.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:23.943+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:24.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:23.944+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:24.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:23.944+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:24.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:23.945+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:24.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:24.310+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:24.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:24.310+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:24.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:24.310+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:24.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:24.311+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:25.265 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:25.042+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:25.265 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:25.042+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:25.265 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:25.042+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:25.266 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:25.043+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:25.266 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:24.911+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:25.266 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:24.911+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:25.266 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:24.911+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:25.266 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:24.912+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:25.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:25.264+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:25.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:25.264+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:25.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:25.265+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:25.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:25.265+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:26.233 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:23:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:23:25.940+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 33402 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:23:26.234 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:25.930+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:26.234 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:25.930+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:26.234 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:25.930+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:26.234 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:25.930+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:26.235 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:26.042+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:26.235 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:26.042+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:26.235 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:26.042+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:26.235 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:26.043+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:26.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:26.233+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:26.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:26.233+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:26.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:26.233+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:26.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:26.234+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:27.247 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:26.902+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:27.247 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:26.902+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:27.247 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:26.902+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:27.248 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:26.903+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:27.248 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:27.069+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:27.248 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:27.069+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:27.248 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:27.069+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:27.248 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:27.070+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:27.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:27.247+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:27.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:27.247+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:27.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:27.247+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:27.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:27.247+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:28.244 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:28.041+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:28.244 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:28.041+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:28.245 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:28.041+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:28.245 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:28.042+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:28.245 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:27.935+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:28.245 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:27.935+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:28.245 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:27.935+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:28.245 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:27.936+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:28.243+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:28.243+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:28.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:28.243+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:28.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:28.244+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:29.219 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:29.082+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:29.220 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:29.082+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:29.220 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:29.082+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:29.220 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:29.082+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:29.220 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:28.956+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:29.220 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:28.957+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:29.220 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:28.957+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:29.220 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:28.958+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:29.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:29.218+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:29.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:29.218+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:29.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:29.218+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:29.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:29.219+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:30.184 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:30.050+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:30.184 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:30.050+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:30.184 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:30.050+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:30.184 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:30.051+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:30.185 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:29.985+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:30.185 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:29.985+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:30.185 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:29.985+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:30.185 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:29.986+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:30.512 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:30.183+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:30.512 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:30.183+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:30.513 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:30.183+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:30.513 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:30.184+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:31.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:23:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:23:30.944+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 33492 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:23:31.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:31.153+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:31.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:31.153+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:31.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:31.153+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:31.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:31.154+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:31.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:30.963+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:31.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:30.963+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:31.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:30.963+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:31.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:30.963+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:31.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:31.026+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:31.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:31.026+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:31.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:31.026+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:31.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:31.027+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:32.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:32.006+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:32.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:32.006+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:32.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:32.006+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:32.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:32.007+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:32.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:31.999+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:32.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:31.999+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:32.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:31.999+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:32.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:32.000+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:32.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:32.135+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:32.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:32.135+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:32.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:32.135+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:32.284 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:32.136+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:33.154 ERROR:paramiko.transport:Socket exception: No route to host (113) 2024-03-24T10:23:33.160 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T10:23:33.160 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-03-24T10:23:33.161 ERROR:teuthology.run_tasks:Manager failed: kclient Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-03-24T10:23:33.174 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-24T10:23:33.188 INFO:tasks.cephadm:Teardown begin 2024-03-24T10:23:33.188 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephadm.py", line 1807, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-03-24T10:23:33.192 DEBUG:teuthology.orchestra.run.smithi022:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T10:23:33.221 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:33.118+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:33.221 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:33.118+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:33.221 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:33.118+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:33.222 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:33.118+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:33.222 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:32.973+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:33.222 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:32.973+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:33.222 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:32.973+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:33.222 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:32.974+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:33.223 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:32.961+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:33.223 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:32.961+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:33.223 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:32.961+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:33.223 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:32.962+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:33.226 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:23:33.226 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:23:34.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:34.126+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:34.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:34.126+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:34.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:34.126+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:34.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:34.126+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:34.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:33.974+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:34.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:33.974+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:34.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:33.974+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:34.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:33.975+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:34.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:33.937+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:34.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:33.937+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:34.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:33.937+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:34.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:33.938+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:35.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:35.146+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:35.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:35.146+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:35.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:35.146+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:35.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:35.147+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:35.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:34.962+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:35.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:34.962+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:35.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:34.962+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:35.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:34.963+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:35.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:34.910+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:35.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:34.910+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:35.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:34.910+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:35.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:34.911+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:36.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:23:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:23:35.949+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 33579 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:23:36.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:36.167+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:36.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:36.167+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:36.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:36.167+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:36.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:36.168+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:36.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:35.992+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:36.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:35.992+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:36.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:35.992+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:36.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:35.993+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:36.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:35.928+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:36.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:35.928+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:36.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:35.928+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:36.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:35.929+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:36.286 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:23:37.216 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:37.003+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:37.217 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:37.003+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:37.217 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:37.003+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:37.217 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:37.004+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:37.217 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:36.926+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:37.217 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:36.926+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:37.217 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:36.926+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:37.217 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:36.927+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:37.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:37.216+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:37.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:37.216+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:37.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:37.216+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:37.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:37.217+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:38.242 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:37.967+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:38.242 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:37.967+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:38.242 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:37.967+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:38.242 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:37.968+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:38.243 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:37.918+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:38.243 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:37.918+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:38.243 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:37.918+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:38.243 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:37.918+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:38.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:38.241+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:38.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:38.241+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:38.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:38.241+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:38.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:38.242+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:39.255 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:38.993+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:39.255 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:38.993+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:39.255 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:38.993+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:39.255 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:38.994+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:39.255 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:38.888+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:39.256 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:38.888+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:39.256 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:38.888+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:39.256 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:38.889+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:39.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:39.254+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:39.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:39.254+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:39.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:39.254+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:39.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:39.255+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:40.211 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:39.967+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:40.212 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:39.967+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:40.212 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:39.967+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:40.212 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:39.968+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:40.212 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:39.899+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:40.213 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:39.899+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:40.213 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:39.899+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:40.213 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:39.900+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:40.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:40.211+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:40.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:40.211+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:40.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:40.211+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:40.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:40.212+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:41.257 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:23:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:23:40.953+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 33669 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:23:41.257 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:40.956+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:41.257 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:40.956+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:41.258 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:40.956+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:41.258 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:40.957+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:41.258 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:40.894+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:41.258 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:40.894+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:41.258 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:40.894+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:41.258 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:40.895+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:41.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:41.256+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:41.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:41.256+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:41.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:41.256+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:41.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:41.257+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:42.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:41.885+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:42.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:41.885+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:42.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:41.885+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:42.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:41.886+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:42.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:41.991+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:42.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:41.991+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:42.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:41.991+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:42.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:41.992+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:42.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:42.293+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:42.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:42.293+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:42.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:42.293+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:42.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:42.294+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:43.276 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:42.887+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:43.276 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:42.887+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:43.276 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:42.887+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:43.277 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:42.888+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:43.277 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:42.991+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:43.277 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:42.991+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:43.277 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:42.991+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:43.277 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:42.992+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:43.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:43.275+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:43.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:43.275+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:43.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:43.275+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:43.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:43.277+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:44.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:44.029+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:44.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:44.029+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:44.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:44.029+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:44.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:44.029+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:44.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:43.901+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:44.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:43.901+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:44.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:43.901+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:44.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:43.902+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:44.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:44.306+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:44.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:44.306+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:44.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:44.306+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:44.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:44.306+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:45.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:44.989+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:45.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:44.989+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:45.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:44.989+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:45.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:44.990+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:45.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:44.913+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:45.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:44.913+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:45.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:44.913+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:45.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:44.913+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:45.287 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:23:45.288 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:23:45.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:45.343+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:45.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:45.343+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:45.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:45.343+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:45.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:45.344+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:46.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:23:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:23:45.958+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 33758 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:23:46.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:45.921+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:46.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:45.921+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:46.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:45.921+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:46.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:45.922+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:46.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:45.962+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:46.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:45.962+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:46.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:45.962+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:46.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:45.963+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:46.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:46.309+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:46.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:46.309+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:46.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:46.309+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:46.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:46.310+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:47.273 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:46.924+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:47.274 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:46.924+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:47.274 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:46.924+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:47.274 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:46.925+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:47.274 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:46.970+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:47.274 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:46.970+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:47.275 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:46.970+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:47.275 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:46.971+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:47.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:47.273+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:47.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:47.273+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:47.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:47.273+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:47.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:47.274+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:48.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:47.905+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:48.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:47.905+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:48.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:47.905+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:48.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:47.906+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:48.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:47.929+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:48.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:47.929+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:48.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:47.929+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:48.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:47.930+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:48.354 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:23:48.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:48.300+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:48.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:48.300+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:48.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:48.300+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:48.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:48.301+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:49.268 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:48.881+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:49.268 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:48.881+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:49.269 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:48.881+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:49.269 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:48.882+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:49.269 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:48.894+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:49.269 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:48.894+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:49.269 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:48.894+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:49.269 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:48.894+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:49.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:49.267+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:49.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:49.267+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:49.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:49.267+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:49.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:49.268+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:50.219 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:49.926+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:50.219 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:49.926+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:50.219 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:49.926+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:50.219 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:49.927+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:50.220 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:49.916+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:50.220 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:49.916+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:50.220 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:49.916+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:50.220 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:49.917+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:50.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:50.286+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:50.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:50.286+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:50.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:50.286+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:50.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:50.287+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:51.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:23:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:23:50.962+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 33849 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:23:51.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:50.938+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:51.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:50.938+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:51.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:50.938+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:51.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:50.939+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:51.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:50.952+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:51.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:50.952+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:51.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:50.952+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:51.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:50.953+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:51.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:51.305+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:51.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:51.305+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:51.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:51.305+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:51.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:51.306+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:52.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:51.934+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:52.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:51.934+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:52.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:51.934+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:52.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:51.935+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:52.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:51.986+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:52.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:51.986+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:52.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:51.986+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:52.285 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:51.987+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:52.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:52.319+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:52.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:52.319+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:52.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:52.319+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:52.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:52.320+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:53.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:52.922+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:53.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:52.922+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:53.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:52.922+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:53.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:52.922+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:53.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:53.005+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:53.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:53.005+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:53.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:53.005+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:53.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:53.006+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:53.650 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:53.286+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:53.650 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:53.286+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:53.650 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:53.286+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:53.650 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:53.287+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:54.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:54.007+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:54.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:54.007+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:54.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:54.007+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:54.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:54.008+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:54.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:53.949+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:54.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:53.949+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:54.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:53.949+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:54.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:53.950+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:54.621 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:54.327+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:54.621 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:54.327+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:54.621 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:54.327+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:54.621 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:54.328+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:55.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:54.983+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:55.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:54.983+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:55.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:54.983+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:55.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:54.984+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:55.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:54.997+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:55.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:54.997+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:55.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:54.997+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:55.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:54.997+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:55.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:55.318+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:55.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:55.318+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:55.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:55.318+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:55.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:55.319+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:56.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:23:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:23:55.966+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 33938 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:23:56.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:55.949+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:56.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:55.949+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:56.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:55.949+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:56.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:55.950+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:56.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:56.028+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:56.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:56.028+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:56.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:56.028+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:56.284 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:56.029+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:56.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:56.357+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:56.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:56.357+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:56.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:56.357+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:56.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:56.357+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:57.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:56.988+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:57.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:56.988+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:57.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:56.988+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:57.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:56.988+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:57.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:57.046+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:57.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:57.046+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:57.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:57.046+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:57.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:57.046+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:57.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:57.387+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:57.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:57.387+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:57.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:57.387+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:57.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:57.388+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:58.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:58.066+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:58.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:58.066+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:58.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:58.066+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:58.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:58.067+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:58.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:57.946+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:58.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:57.946+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:58.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:57.946+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:58.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:57.947+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:58.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:58.365+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:58.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:58.365+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:58.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:58.365+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:58.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:58.366+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:59.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:59.073+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:59.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:59.073+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:59.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:59.073+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:23:59.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:23:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:23:59.073+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:59.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:58.946+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:59.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:58.946+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:59.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:58.946+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:23:59.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:58.947+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:23:59.660 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:59.331+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:59.660 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:59.331+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:59.660 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:59.331+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:23:59.661 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:23:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:23:59.332+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:00.220 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:59.957+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:00.220 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:59.957+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:00.221 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:59.957+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:00.221 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:23:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:23:59.959+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:00.221 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:00.100+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:00.221 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:00.100+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:00.221 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:00.100+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:00.221 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:00.101+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:00.355 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:24:00.356 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:24:00.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:00.353+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:00.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:00.353+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:00.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:00.353+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:00.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:00.354+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:01.232 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:24:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:24:00.970+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 34028 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:24:01.233 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:01.126+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:01.233 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:01.126+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:01.233 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:01.126+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:01.233 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:01.126+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:01.234 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:00.965+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:01.234 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:00.965+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:01.234 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:00.965+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:01.234 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:00.966+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:01.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:01.400+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:01.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:01.400+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:01.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:01.400+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:01.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:01.401+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:02.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:02.153+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:02.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:02.153+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:02.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:02.153+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:02.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:02.154+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:02.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:01.980+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:02.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:01.980+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:02.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:01.980+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:02.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:01.980+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:02.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:02.392+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:02.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:02.392+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:02.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:02.392+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:02.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:02.393+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:03.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:03.105+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:03.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:03.105+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:03.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:03.105+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:03.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:03.105+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:03.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:02.968+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:03.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:02.968+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:03.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:02.968+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:03.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:02.969+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:03.423 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:24:03.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:03.434+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:03.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:03.434+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:03.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:03.434+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:03.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:03.435+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:04.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:04.131+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:04.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:04.131+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:04.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:04.131+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:04.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:04.132+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:04.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:03.995+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:04.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:03.995+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:04.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:03.995+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:04.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:03.996+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:04.700 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:04.465+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:04.701 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:04.465+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:04.701 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:04.465+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:04.701 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:04.466+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:05.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:05.155+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:05.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:05.155+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:05.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:05.155+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:05.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:05.156+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:05.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:04.977+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:05.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:04.977+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:05.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:04.977+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:05.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:04.978+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:05.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:05.478+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:05.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:05.478+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:05.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:05.478+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:05.783 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:05.478+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:06.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:24:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:24:05.975+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 34111 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:24:06.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:06.201+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:06.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:06.201+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:06.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:06.201+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:06.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:06.202+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:06.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:05.958+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:06.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:05.958+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:06.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:05.958+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:06.284 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:05.959+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:06.424 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:24:06.425 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:24:06.494 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:24:06.494 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-24T10:24:06.495 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-24T10:24:06.495 DEBUG:teuthology.orchestra.run.smithi022:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T10:24:06.510 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:24:06.511 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:24:06.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:06.505+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:06.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:06.505+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:06.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:06.505+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:06.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:06.506+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:07.210 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:06.947+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:07.211 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:06.947+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:07.211 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:06.947+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:07.211 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:06.947+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:07.525 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:07.211+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:07.525 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:07.211+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:07.525 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:07.211+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:07.526 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:07.212+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:07.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:07.526+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:07.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:07.526+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:07.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:07.526+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:07.782 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:07.526+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:08.212 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:07.937+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:08.213 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:07.937+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:08.213 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:07.937+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:08.213 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:07.938+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:08.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:08.213+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:08.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:08.213+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:08.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:08.213+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:08.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:08.214+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:08.924 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:08.547+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:08.924 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:08.547+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:08.925 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:08.547+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:08.925 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:08.548+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:09.233 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:08.925+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:09.234 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:08.925+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:09.234 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:08.925+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:09.234 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:08.926+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:09.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:09.234+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:09.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:09.234+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:09.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:09.234+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:09.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:09.235+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:09.566 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:24:09.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:09.592+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:09.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:09.592+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:09.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:09.592+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:09.854 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:09.593+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:10.221 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:09.905+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:10.221 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:09.905+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:10.221 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:09.905+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:10.221 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:09.907+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:10.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:10.232+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:10.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:10.232+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:10.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:10.232+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:10.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:10.232+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:10.865 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:10.591+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:10.865 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:10.591+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:10.865 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:10.591+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:10.865 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:10.592+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:11.236 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:24:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:24:10.980+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 34201 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:24:11.236 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:10.865+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:11.237 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:10.865+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:11.237 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:10.865+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:11.237 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:10.866+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:11.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:11.236+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:11.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:11.237+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:11.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:11.237+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:11.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:11.238+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:11.834 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:11.544+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:11.835 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:11.544+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:11.835 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:11.544+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:11.835 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:11.544+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:12.256 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:11.835+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:12.256 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:11.835+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:12.257 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:11.835+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:12.257 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:11.836+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:12.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:12.257+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:12.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:12.257+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:12.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:12.257+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:12.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:12.258+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:13.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:12.583+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:13.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:12.583+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:13.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:12.583+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:13.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:12.584+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:13.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:12.791+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:13.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:12.791+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:13.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:12.791+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:13.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:12.792+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:13.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:13.241+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:13.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:13.241+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:13.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:13.241+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:13.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:13.242+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:14.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:13.627+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:14.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:13.627+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:14.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:13.627+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:14.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:13.628+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:14.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:13.760+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:14.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:13.760+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:14.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:13.760+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:14.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:13.760+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:14.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:14.274+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:14.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:14.274+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:14.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:14.274+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:14.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:14.275+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:15.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:14.654+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:15.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:14.654+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:15.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:14.654+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:15.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:14.655+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:15.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:14.716+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:15.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:14.716+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:15.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:14.716+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:15.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:14.716+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:15.670 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:15.302+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:15.670 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:15.302+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:15.670 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:15.302+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:15.671 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:15.303+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:15.984 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:15.669+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:15.984 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:15.669+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:15.984 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:15.669+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:15.984 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:15.671+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:15.984 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:15.679+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:15.985 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:15.679+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:15.985 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:15.679+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:15.985 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:15.680+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:16.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:24:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:24:15.983+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 34289 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:24:16.593 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:16.304+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:16.593 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:16.304+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:16.593 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:16.304+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:16.593 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:16.305+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:17.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:16.705+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:17.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:16.705+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:17.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:16.705+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:17.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:16.706+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:17.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:16.694+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:17.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:16.694+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:17.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:16.694+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:17.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:16.695+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:17.694 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:17.354+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:17.694 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:17.354+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:17.694 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:17.354+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:17.694 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:17.355+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:18.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:17.693+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:18.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:17.693+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:18.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:17.693+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:18.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:17.694+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:18.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:17.702+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:18.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:17.702+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:18.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:17.702+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:18.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:17.702+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:18.567 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:24:18.568 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:24:18.656 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:18.309+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:18.656 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:18.309+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:18.656 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:18.309+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:18.656 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:18.310+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:19.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:18.655+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:19.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:18.655+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:19.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:18.655+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:19.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:18.656+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:19.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:18.672+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:19.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:18.672+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:19.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:18.672+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:19.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:18.672+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:19.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:19.274+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:19.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:19.274+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:19.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:19.274+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:19.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:19.275+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:20.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:19.628+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:20.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:19.628+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:20.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:19.628+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:20.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:19.628+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:20.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:19.676+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:20.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:19.676+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:20.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:19.676+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:20.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:19.677+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:20.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:20.305+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:20.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:20.305+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:20.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:20.305+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:20.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:20.306+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:20.989 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:20.650+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:20.989 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:20.650+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:20.989 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:20.650+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:20.989 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:20.651+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:20.989 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:20.689+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:20.990 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:20.689+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:20.990 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:20.689+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:20.990 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:20.690+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:21.263 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:24:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:24:20.987+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 34380 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:24:21.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:21.280+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:21.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:21.280+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:21.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:21.280+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:21.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:21.281+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:21.630 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:24:22.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:21.656+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:22.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:21.656+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:22.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:21.656+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:22.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:21.657+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:22.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:21.672+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:22.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:21.672+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:22.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:21.672+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:22.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:21.673+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:22.659 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:22.321+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:22.659 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:22.321+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:22.659 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:22.321+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:22.659 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:22.321+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:23.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:22.658+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:23.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:22.658+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:23.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:22.658+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:23.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:22.659+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:23.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:22.702+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:23.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:22.702+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:23.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:22.702+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:23.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:22.703+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:23.652 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:23.352+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:23.653 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:23.352+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:23.653 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:23.352+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:23.653 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:23.353+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:24.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:23.694+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:24.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:23.694+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:24.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:23.694+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:24.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:23.695+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:24.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:23.686+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:24.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:23.686+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:24.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:23.686+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:24.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:23.687+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:24.715 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:24.387+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:24.716 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:24.387+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:24.716 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:24.387+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:24.716 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:24.388+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:25.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:24.715+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:25.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:24.715+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:25.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:24.715+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:25.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:24.715+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:25.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:24.731+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:25.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:24.731+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:25.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:24.731+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:25.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:24.731+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:25.700 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:25.344+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:25.700 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:25.344+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:25.701 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:25.344+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:25.701 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:25.345+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:25.992 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:25.699+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:25.993 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:25.699+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:25.993 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:25.700+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:25.993 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:25.700+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:25.993 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:25.706+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:25.993 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:25.706+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:25.994 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:25.706+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:25.994 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:25.707+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:26.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:24:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:24:25.992+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 34469 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:24:26.665 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:26.335+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:26.666 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:26.335+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:26.666 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:26.335+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:26.666 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:26.336+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:27.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:26.665+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:27.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:26.665+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:27.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:26.665+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:27.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:26.665+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:27.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:26.737+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:27.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:26.737+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:27.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:26.737+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:27.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:26.738+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:27.665 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:27.367+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:27.665 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:27.367+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:27.665 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:27.367+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:27.665 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:27.368+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:28.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:27.664+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:28.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:27.664+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:28.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:27.664+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:28.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:27.665+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:28.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:27.765+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:28.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:27.765+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:28.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:27.765+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:28.035 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:27.766+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:28.660 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:28.404+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:28.660 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:28.404+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:28.660 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:28.404+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:28.660 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:28.405+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:29.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:28.658+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:29.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:28.658+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:29.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:28.658+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:29.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:28.659+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:29.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:28.800+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:29.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:28.800+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:29.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:28.800+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:29.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:28.801+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:29.613 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:29.359+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:29.614 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:29.359+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:29.614 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:29.359+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:29.614 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:29.360+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:29.910 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:29.613+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:29.910 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:29.613+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:29.911 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:29.613+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:29.911 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:29.614+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:29.911 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:29.824+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:29.911 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:29.824+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:29.911 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:29.824+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:29.911 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:29.825+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:30.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:30.349+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:30.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:30.349+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:30.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:30.349+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:30.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:30.350+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:30.997 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:30.783+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:30.997 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:30.783+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:30.997 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:30.783+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:30.997 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:30.784+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:30.998 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:30.651+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:30.998 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:30.651+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:30.998 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:30.651+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:30.998 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:30.652+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:31.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:24:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:24:30.996+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 34558 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:24:31.626 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:31.315+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:31.626 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:31.315+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:31.626 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:31.315+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:31.627 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:31.316+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:31.952 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:31.625+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:31.953 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:31.625+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:31.953 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:31.625+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:31.953 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:31.626+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:31.953 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:31.811+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:31.953 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:31.811+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:31.954 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:31.811+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:31.954 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:31.811+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:32.675 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:32.319+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:32.675 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:32.319+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:32.675 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:32.319+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:32.675 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:32.320+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:33.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:32.799+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:33.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:32.799+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:33.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:32.799+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:33.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:32.800+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:33.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:32.674+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:33.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:32.674+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:33.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:32.674+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:33.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:32.675+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:33.632 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:24:33.632 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:24:33.653 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:33.307+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:33.653 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:33.307+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:33.653 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:33.307+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:33.654 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:33.308+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:34.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:33.699+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:34.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:33.699+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:34.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:33.699+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:34.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:33.700+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:34.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:33.783+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:34.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:33.783+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:34.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:33.783+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:34.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:33.784+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:34.749 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:34.344+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:34.749 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:34.344+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:34.749 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:34.344+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:34.749 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:34.345+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:35.024 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:34.811+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:35.025 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:34.811+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:35.025 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:34.811+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:35.025 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:34.812+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:35.025 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:34.748+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:35.025 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:34.748+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:35.026 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:34.748+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:35.026 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:34.749+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:35.727 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:35.350+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:35.727 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:35.350+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:35.727 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:35.350+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:35.727 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:35.351+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:36.001 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:35.726+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:36.002 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:35.726+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:36.002 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:35.726+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:36.002 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:35.727+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:36.002 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:35.765+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:36.002 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:35.765+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:36.002 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:35.765+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:36.003 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:35.766+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:36.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:24:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:24:36.000+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 34648 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:24:36.702 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:24:36.728 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:36.366+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:36.728 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:36.366+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:36.728 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:36.366+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:36.728 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:36.367+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:37.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:36.747+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:37.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:36.747+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:37.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:36.747+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:37.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:36.749+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:37.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:36.727+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:37.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:36.727+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:37.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:36.727+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:37.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:36.728+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:37.717 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:37.345+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:37.717 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:37.345+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:37.717 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:37.345+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:37.718 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:37.346+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:38.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:37.716+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:38.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:37.716+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:38.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:37.716+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:38.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:37.717+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:38.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:37.751+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:38.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:37.751+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:38.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:37.751+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:38.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:37.752+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:38.734 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:38.321+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:38.734 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:38.321+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:38.735 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:38.321+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:38.735 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:38.321+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:38.767+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:38.767+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:38.767+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:39.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:38.768+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:39.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:38.733+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:39.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:38.733+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:39.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:38.733+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:39.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:38.734+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:39.704 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:24:39.704 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:24:39.766 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:39.288+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:39.766 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:39.288+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:39.766 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:39.288+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:39.767 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:39.289+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:39.774 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:24:39.774 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-24T10:24:39.775 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-24T10:24:39.776 DEBUG:teuthology.misc:Transferring archived files from smithi022:/var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619043/remote/smithi022/crash 2024-03-24T10:24:39.777 DEBUG:teuthology.orchestra.run.smithi022:> sudo tar c -f - -C /var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/crash -- . 2024-03-24T10:24:39.840 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/lib/ceph/45356448-e9c2-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619043/remote/smithi155/crash 2024-03-24T10:24:39.841 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:24:39.842 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:24:40.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:39.765+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:40.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:39.765+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:40.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:39.765+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:40.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:39.766+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:40.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:39.778+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:40.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:39.778+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:40.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:39.778+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:40.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:39.778+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:40.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:40.252+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:40.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:40.252+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:40.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:40.252+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:40.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:40.253+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:41.006 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:40.781+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:41.006 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:40.781+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:41.006 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:40.781+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:41.006 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:40.782+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:41.006 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:40.737+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:41.007 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:40.737+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:41.007 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:40.737+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:41.007 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:40.738+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:41.270 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:24:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:24:41.005+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 34738 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:24:41.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:41.269+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:41.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:41.269+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:41.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:41.269+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:41.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:41.270+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:42.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:41.814+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:42.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:41.814+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:42.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:41.814+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:42.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:41.815+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:42.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:41.689+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:42.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:41.689+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:42.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:41.689+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:42.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:41.690+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:42.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:42.251+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:42.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:42.251+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:42.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:42.251+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:42.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:42.251+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:42.910 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:24:43.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:42.774+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:43.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:42.774+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:43.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:42.774+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:43.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:42.775+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:43.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:42.692+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:43.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:42.692+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:43.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:42.692+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:43.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:42.693+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:43.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:43.235+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:43.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:43.235+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:43.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:43.235+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:43.534 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:43.235+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:44.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:43.727+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:44.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:43.727+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:44.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:43.727+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:44.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:43.728+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:44.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:43.709+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:44.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:43.709+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:44.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:43.709+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:44.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:43.711+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:44.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:44.217+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:44.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:44.217+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:44.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:44.217+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:44.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:44.217+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:45.030 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:44.760+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:45.031 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:44.760+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:45.031 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:44.760+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:45.031 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:44.760+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:45.031 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:44.737+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:45.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:44.737+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:45.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:44.737+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:45.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:44.738+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:45.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:45.173+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:45.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:45.173+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:45.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:45.173+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:45.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:45.174+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:46.010 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:45.755+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:46.011 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:45.755+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:46.011 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:45.755+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:46.011 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:45.756+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:46.011 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:45.729+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:46.011 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:45.729+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:46.012 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:45.729+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:46.012 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:45.729+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:46.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:24:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:24:46.010+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 34829 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:24:46.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:46.184+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:46.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:46.184+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:46.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:46.184+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:46.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:46.185+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:47.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:46.721+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:47.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:46.721+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:47.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:46.721+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:47.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:46.722+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:47.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:46.750+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:47.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:46.750+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:47.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:46.750+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:47.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:46.751+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:47.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:47.226+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:47.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:47.226+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:47.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:47.226+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:47.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:47.226+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:48.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:47.794+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:48.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:47.794+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:48.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:47.794+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:48.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:47.795+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:48.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:47.740+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:48.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:47.740+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:48.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:47.740+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:48.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:47.740+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:48.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:48.255+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:48.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:48.255+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:48.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:48.255+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:48.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:48.256+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:49.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:48.742+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:49.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:48.742+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:49.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:48.742+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:49.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:48.742+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:49.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:48.805+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:49.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:48.805+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:49.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:48.805+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:49.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:48.806+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:49.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:49.305+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:49.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:49.305+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:49.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:49.305+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:49.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:49.306+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:50.073 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:49.791+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:50.074 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:49.791+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:50.074 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:49.791+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:50.074 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:49.792+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:50.074 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:49.823+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:50.074 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:49.823+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:50.074 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:49.823+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:50.075 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:49.824+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:50.384 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:50.256+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:50.384 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:50.256+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:50.385 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:50.256+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:50.385 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:50.258+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:51.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:24:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:24:51.014+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 34915 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:24:51.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:50.873+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:51.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:50.873+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:51.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:50.873+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:51.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:50.874+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:51.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:50.780+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:51.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:50.780+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:51.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:50.780+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:51.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:50.780+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:51.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:51.212+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:51.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:51.212+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:51.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:51.212+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:51.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:51.212+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:51.913 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:24:51.914 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:24:52.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:51.753+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:52.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:51.753+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:52.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:51.753+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:52.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:51.754+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:52.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:51.903+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:52.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:51.903+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:52.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:51.903+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:52.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:51.904+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:52.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:52.174+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:52.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:52.174+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:52.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:52.174+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:52.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:52.176+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:53.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:52.913+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:53.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:52.913+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:53.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:52.913+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:53.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:52.914+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:53.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:52.761+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:53.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:52.761+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:53.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:52.761+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:53.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:52.762+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:53.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:53.203+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:53.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:53.203+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:53.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:53.203+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:53.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:53.204+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:54.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:53.772+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:54.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:53.772+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:54.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:53.772+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:54.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:53.773+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:54.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:53.866+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:54.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:53.866+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:54.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:53.866+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:54.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:53.866+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:54.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:54.209+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:54.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:54.209+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:54.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:54.209+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:54.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:54.210+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:54.974 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:24:55.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:54.891+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:55.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:54.891+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:55.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:54.891+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:55.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:54.892+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:55.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:54.731+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:55.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:54.731+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:55.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:54.731+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:55.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:54.732+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:55.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:55.207+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:55.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:55.207+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:55.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:55.207+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:55.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:55.208+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:56.019 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:55.736+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:56.019 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:55.736+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:56.019 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:55.736+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:56.020 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:55.737+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:56.020 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:55.911+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:56.020 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:55.911+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:56.020 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:55.912+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:56.020 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:55.912+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:56.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:24:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:24:56.018+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 35005 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:24:56.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:56.196+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:56.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:56.196+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:56.282 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:56.196+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:56.283 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:56.197+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:57.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:56.888+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:57.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:56.888+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:57.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:56.888+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:57.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:56.889+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:57.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:56.778+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:57.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:56.778+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:57.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:56.778+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:57.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:56.779+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:57.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:57.218+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:57.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:57.218+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:57.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:57.218+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:57.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:57.219+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:58.257 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:57.919+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:58.257 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:57.919+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:58.257 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:57.919+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:58.257 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:57.920+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:58.258 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:57.801+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:58.258 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:57.801+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:58.258 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:57.801+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:58.258 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:57.801+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:58.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:58.256+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:58.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:58.256+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:58.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:58.256+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:58.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:58.258+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:59.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:58.828+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:59.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:58.828+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:59.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:58.828+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:24:59.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:58.829+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:59.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:58.968+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:59.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:58.968+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:59.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:58.968+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:24:59.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:58.968+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:24:59.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:59.285+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:59.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:59.285+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:59.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:59.285+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:24:59.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:24:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:24:59.285+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:00.153 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:59.810+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:00.154 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:59.810+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:00.154 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:59.810+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:00.154 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:24:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:24:59.810+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:00.154 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:59.923+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:00.154 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:59.923+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:00.155 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:59.923+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:00.155 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:24:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:24:59.924+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:00.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:00.267+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:00.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:00.267+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:00.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:00.267+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:00.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:00.268+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:01.024 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:00.768+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:01.024 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:00.768+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:01.024 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:00.768+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:01.024 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:00.769+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:01.025 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:00.939+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:01.025 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:00.939+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:01.025 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:00.939+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:01.025 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:00.940+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:01.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:25:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:25:01.023+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 35095 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:25:01.742 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:01.284+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:01.742 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:01.284+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:01.742 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:01.284+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:01.743 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:01.285+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:02.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:01.930+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:02.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:01.930+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:02.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:01.930+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:02.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:01.930+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:02.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:01.741+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:02.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:01.741+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:02.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:01.741+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:02.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:01.742+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:02.672 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:02.329+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:02.673 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:02.329+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:02.673 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:02.329+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:02.673 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:02.330+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:02.967 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:02.728+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:02.968 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:02.728+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:02.968 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:02.728+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:02.968 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:02.729+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:03.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:02.966+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:03.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:02.966+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:03.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:02.966+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:03.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:02.967+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:03.654 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:03.362+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:03.655 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:03.362+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:03.655 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:03.362+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:03.655 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:03.363+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:03.952 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:03.711+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:03.952 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:03.711+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:03.953 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:03.711+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:03.953 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:03.711+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:04.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:03.951+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:04.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:03.951+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:04.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:03.951+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:04.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:03.952+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:04.760 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:04.389+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:04.760 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:04.389+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:04.760 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:04.389+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:04.761 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:04.389+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:05.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:04.759+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:05.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:04.759+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:05.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:04.759+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:05.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:04.761+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:05.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:04.913+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:05.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:04.913+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:05.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:04.913+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:05.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:04.913+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:05.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:05.371+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:05.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:05.371+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:05.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:05.371+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:05.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:05.372+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:06.028 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:05.946+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:06.029 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:05.946+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:06.029 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:05.946+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:06.029 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:05.946+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:06.029 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:05.776+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:06.029 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:05.776+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:06.030 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:05.776+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:06.030 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:05.777+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:06.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:25:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:25:06.027+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 35186 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:25:06.738 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:06.410+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:06.739 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:06.410+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:06.739 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:06.410+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:06.739 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:06.411+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:06.975 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:25:06.976 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:25:06.993 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:06.738+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:06.994 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:06.738+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:06.994 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:06.738+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:06.994 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:06.739+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:07.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:06.993+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:07.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:06.993+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:07.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:06.993+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:07.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:06.994+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:07.715 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:07.439+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:07.715 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:07.439+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:07.715 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:07.439+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:07.716 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:07.440+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:08.019 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:07.714+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:08.019 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:07.714+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:08.019 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:07.714+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:08.020 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:07.715+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:08.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:08.019+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:08.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:08.019+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:08.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:08.019+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:08.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:08.019+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:08.754 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:08.487+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:08.754 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:08.487+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:08.754 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:08.487+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:08.754 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:08.488+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:09.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:08.974+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:09.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:08.974+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:09.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:08.974+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:09.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:08.975+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:09.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:08.753+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:09.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:08.753+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:09.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:08.753+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:09.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:08.754+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:09.765 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:09.443+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:09.765 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:09.443+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:09.765 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:09.443+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:09.765 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:09.444+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:10.020 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:09.764+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:10.020 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:09.764+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:10.020 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:09.764+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:10.020 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:09.765+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:10.046 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:25:10.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:10.019+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:10.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:10.019+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:10.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:10.019+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:10.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:10.020+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:10.738 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:10.464+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:10.738 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:10.464+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:10.738 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:10.464+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:10.738 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:10.464+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:11.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:10.737+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:11.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:10.737+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:11.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:10.737+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:11.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:10.738+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:11.474 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:25:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:25:11.032+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 35275 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:25:11.475 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:11.038+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:11.475 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:11.038+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:11.475 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:11.038+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:11.475 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:11.039+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:11.765 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:11.474+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:11.766 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:11.474+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:11.766 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:11.474+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:11.766 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:11.475+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:12.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:11.765+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:12.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:11.765+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:12.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:11.765+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:12.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:11.765+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:12.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:11.999+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:12.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:11.999+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:12.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:11.999+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:12.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:12.000+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:12.763 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:12.469+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:12.764 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:12.469+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:12.764 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:12.469+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:12.764 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:12.469+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:13.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:12.972+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:13.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:12.972+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:13.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:12.972+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:13.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:12.972+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:13.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:12.763+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:13.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:12.763+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:13.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:12.763+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:13.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:12.763+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:13.047 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:25:13.048 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:25:13.118 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:25:13.118 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-24T10:25:13.119 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-24T10:25:13.119 DEBUG:teuthology.orchestra.run.smithi022:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-24T10:25:13.152 INFO:tasks.cephadm:Compressing logs... 2024-03-24T10:25:13.153 DEBUG:teuthology.orchestra.run.smithi022:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T10:25:13.195 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:25:13.196 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:25:13.219 INFO:teuthology.orchestra.run.smithi022.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T10:25:13.219 INFO:teuthology.orchestra.run.smithi022.stderr:: No such file or directory 2024-03-24T10:25:13.220 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T10:25:13.220 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph-osd.1.log 2024-03-24T10:25:13.220 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph-mon.smithi022.log 2024-03-24T10:25:13.220 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph.audit.log: /var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph-mds.cephfs.smithi022.qbmomh.log 2024-03-24T10:25:13.221 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph-mon.smithi022.log: gzip -5 --verbose -- /var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph-mgr.smithi022.emgmxz.log 2024-03-24T10:25:13.221 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph-mds.cephfs.smithi022.qbmomh.log: gzip -5 --verbose -- /var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph-osd.0.log 2024-03-24T10:25:13.222 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph-mgr.smithi022.emgmxz.log: gzip -5 --verbose -- /var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T10:25:13.222 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph-osd.0.log: 90.6% -- replaced with /var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T10:25:13.227 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph.log 2024-03-24T10:25:13.228 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph.cephadm.log: 83.4% -- replaced with /var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T10:25:13.231 INFO:teuthology.orchestra.run.smithi022.stderr: 90.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T10:25:13.231 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph-osd.2.log 2024-03-24T10:25:13.232 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T10:25:13.233 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph-osd.2.log: 86.7% -- replaced with /var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T10:25:13.233 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph-client.ceph-exporter.smithi022.log 2024-03-24T10:25:13.245 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph-volume.log: 90.9% -- replaced with /var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T10:25:13.245 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph-mds.cephfs.smithi022.mhcbrm.log 2024-03-24T10:25:13.249 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph-client.ceph-exporter.smithi022.log: 94.6% -- replaced with /var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph-client.ceph-exporter.smithi022.log.gz 2024-03-24T10:25:13.292 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph-mds.cephfs.smithi022.mhcbrm.log: gzip: /var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph-mds.cephfs.smithi022.mhcbrm.log: file size changed while zipping 2024-03-24T10:25:13.292 INFO:teuthology.orchestra.run.smithi022.stderr: 95.0% -- replaced with /var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph-mds.cephfs.smithi022.mhcbrm.log.gz 2024-03-24T10:25:13.769 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:13.438+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:13.770 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:13.438+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:13.770 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:13.438+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:13.770 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:13.439+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:14.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:13.768+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:14.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:13.768+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:14.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:13.768+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:14.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:13.770+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:14.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:13.980+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:14.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:13.980+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:14.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:13.980+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:14.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:13.981+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:14.110 INFO:teuthology.orchestra.run.smithi022.stderr:gzip: /var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph-mgr.smithi022.emgmxz.log: file size changed while zipping 2024-03-24T10:25:14.110 INFO:teuthology.orchestra.run.smithi022.stderr: 89.9% -- replaced with /var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph-mgr.smithi022.emgmxz.log.gz 2024-03-24T10:25:14.637 INFO:teuthology.orchestra.run.smithi022.stderr:gzip: /var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph-mon.smithi022.log: file size changed while zipping 2024-03-24T10:25:14.638 INFO:teuthology.orchestra.run.smithi022.stderr: 92.7% -- replaced with /var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph-mon.smithi022.log.gz 2024-03-24T10:25:14.770 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:14.395+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:14.770 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:14.395+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:14.770 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:14.395+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:14.770 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:14.396+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:15.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:14.769+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:15.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:14.769+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:15.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:14.769+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:15.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:14.770+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:15.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:14.970+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:15.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:14.970+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:15.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:14.970+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:15.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:14.971+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:15.116 INFO:teuthology.orchestra.run.smithi022.stderr: 95.3% -- replaced with /var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph-mds.cephfs.smithi022.qbmomh.log.gz 2024-03-24T10:25:15.749 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:15.375+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:15.749 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:15.375+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:15.749 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:15.375+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:15.750 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:15.376+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:16.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:15.748+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:16.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:15.748+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:16.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:15.748+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:16.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:15.750+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:16.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:15.991+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:16.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:15.991+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:16.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:15.991+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:16.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:15.992+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:16.254 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:25:16.373 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:25:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:25:16.036+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 35365 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:25:16.614 INFO:teuthology.orchestra.run.smithi022.stderr:gzip: /var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph-osd.2.log: file size changed while zipping 2024-03-24T10:25:16.614 INFO:teuthology.orchestra.run.smithi022.stderr: 91.1% -- replaced with /var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph-osd.2.log.gz 2024-03-24T10:25:16.728 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:16.373+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:16.729 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:16.373+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:16.729 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:16.373+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:16.729 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:16.374+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:17.004 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:16.728+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:17.005 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:16.728+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:17.005 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:16.728+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:17.005 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:16.729+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:17.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:17.004+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:17.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:17.004+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:17.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:17.004+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:17.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:17.005+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:17.692 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:17.414+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:17.692 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:17.414+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:17.692 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:17.414+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:17.693 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:17.415+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:18.016 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:17.691+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:18.016 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:17.691+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:18.017 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:17.691+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:18.017 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:17.693+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:18.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:18.016+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:18.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:18.016+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:18.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:18.016+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:18.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:18.016+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:18.720 INFO:teuthology.orchestra.run.smithi022.stderr:gzip: /var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph-osd.0.log: file size changed while zipping 2024-03-24T10:25:18.720 INFO:teuthology.orchestra.run.smithi022.stderr: 94.9% -- replaced with /var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph-osd.0.log.gz 2024-03-24T10:25:18.785 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:18.665+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:18.785 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:18.665+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:18.785 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:18.665+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:18.785 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:18.666+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:18.786 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:18.429+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:18.786 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:18.429+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:18.786 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:18.429+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:18.786 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:18.430+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:19.055 INFO:teuthology.orchestra.run.smithi022.stderr:gzip: /var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph-osd.1.log: file size changed while zipping 2024-03-24T10:25:19.055 INFO:teuthology.orchestra.run.smithi022.stderr: 94.8% -- replaced with /var/log/ceph/45356448-e9c2-11ee-95cd-87774f69a715/ceph-osd.1.log.gz 2024-03-24T10:25:19.057 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-24T10:25:19.057 INFO:teuthology.orchestra.run.smithi022.stderr:real 0m5.850s 2024-03-24T10:25:19.057 INFO:teuthology.orchestra.run.smithi022.stderr:user 0m18.578s 2024-03-24T10:25:19.057 INFO:teuthology.orchestra.run.smithi022.stderr:sys 0m0.381s 2024-03-24T10:25:19.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:18.969+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:19.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:18.969+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:19.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:18.969+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:19.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:18.970+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:19.658 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:19.407+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:19.658 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:19.407+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:19.658 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:19.407+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:19.659 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:19.407+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:19.659 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:19.657+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:19.659 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:19.657+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:19.659 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:19.657+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:19.659 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:19.657+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:20.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:19.997+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:20.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:19.997+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:20.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:19.997+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:20.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:19.998+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:20.663 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:20.449+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:20.663 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:20.449+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:20.663 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:20.449+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:20.664 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:20.449+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:21.026 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:20.662+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:21.026 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:20.662+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:21.026 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:20.662+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:21.026 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:20.663+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:21.282 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:25:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:25:21.041+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 35454 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:25:21.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:21.025+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:21.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:21.025+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:21.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:21.025+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:21.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:21.026+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:21.668 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:21.401+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:21.668 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:21.401+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:21.668 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:21.401+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:21.668 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:21.401+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:22.025 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:21.667+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:22.025 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:21.667+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:22.025 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:21.667+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:22.025 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:21 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:21.668+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:22.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:22.024+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:22.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:22.024+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:22.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:22.024+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:22.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:22.025+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:22.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:22.630+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:22.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:22.630+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:22.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:22.630+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:22.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:22.630+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:22.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:22.395+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:22.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:22.395+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:22.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:22.395+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:22.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:22 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:22.395+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:23.402 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:23.072+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:23.403 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:23.072+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:23.403 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:23.072+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:23.403 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:23.073+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:23.656 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:23.402+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:23.656 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:23.402+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:23.657 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:23.402+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:23.657 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:23.403+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:23.657 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:23.643+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:23.657 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:23.643+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:23.657 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:23.643+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:23.658 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:23 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:23.644+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:24.413 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:24.091+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:24.414 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:24.091+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:24.414 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:24.091+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:24.414 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:24.092+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:24.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:24.414+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:24.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:24.414+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:24.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:24.414+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:24.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:24.415+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:24.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:24.653+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:24.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:24.653+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:24.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:24.653+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:24.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:24 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:24.654+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:25.255 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:25:25.256 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:25:25.364 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:25.091+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:25.364 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:25.091+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:25.364 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:25.091+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:25.365 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:25.092+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:25.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:25.609+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:25.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:25.609+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:25.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:25.609+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:25.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:25.610+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:25.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:25.435+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:25.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:25.435+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:25.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:25.435+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:25.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:25 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:25.436+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:26.396 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:25:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:25:26.047+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 35545 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:25:26.397 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:26.120+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:26.397 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:26.120+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:26.397 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:26.120+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:26.397 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:26.121+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:26.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:26.397+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:26.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:26.397+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:26.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:26.397+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:26.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:26.398+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:26.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:26.643+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:26.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:26.643+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:26.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:26.643+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:26.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:26 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:26.644+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:27.433 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:27.097+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:27.433 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:27.097+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:27.433 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:27.098+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:27.433 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:27.098+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:27.683 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:27.683+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:27.683 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:27.433+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:27.683 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:27.433+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:27.684 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:27.433+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:27.684 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:27.434+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:28.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:27.683+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:28.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:27.683+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:28.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:27 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:27.684+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:28.322 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:25:28.480 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:28.121+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:28.480 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:28.121+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:28.480 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:28.121+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:28.481 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:28.121+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:28.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:28.481+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:28.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:28.481+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:28.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:28.481+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:28.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:28.481+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:28.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:28.696+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:28.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:28.696+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:28.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:28.696+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:28.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:28 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:28.697+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:29.506 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:29.109+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:29.506 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:29.109+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:29.506 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:29.109+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:29.507 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:29.109+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:29.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:29.726+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:29.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:29.726+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:29.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:29.726+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:29.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:29.727+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:29.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:29.506+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:29.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:29.506+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:29.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:29.506+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:29.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:29 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:29.507+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:30.225 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:30.092+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:30.225 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:30.092+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:30.225 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:30.092+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:30.225 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:30.093+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:30.769 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:30.494+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:30.769 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:30.494+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:30.769 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:30.494+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:30.769 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:30.495+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:31.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:30.769+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:31.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:30.769+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:31.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:30.769+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:31.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:30 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:30.770+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:31.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:25:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:25:31.051+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 35635 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:25:31.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:31.118+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:31.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:31.118+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:31.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:31.118+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:31.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:31.119+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:32.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:31.534+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:32.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:31.534+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:32.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:31.534+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:32.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:31.534+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:32.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:31.756+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:32.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:31.756+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:32.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:31.756+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:32.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:31 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:31.757+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:32.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:32.104+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:32.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:32.104+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:32.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:32.104+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:32.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:32.105+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:33.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:32.582+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:33.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:32.582+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:33.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:32.582+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:33.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:32.582+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:33.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:32.798+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:33.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:32.798+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:33.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:32.798+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:33.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:32 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:32.798+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:33.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:33.100+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:33.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:33.100+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:33.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:33.100+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:33.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:33.100+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:34.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:33.762+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:34.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:33.762+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:34.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:33.762+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:34.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:33.763+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:34.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:33.591+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:34.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:33.591+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:34.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:33.591+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:34.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:33 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:33.591+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:34.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:34.118+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:34.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:34.118+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:34.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:34.118+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:34.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:34.119+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:35.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:34.556+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:35.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:34.556+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:35.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:34.556+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:35.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:34.556+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:35.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:34.801+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:35.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:34.801+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:35.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:34.801+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:35.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:34 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:34.802+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:35.448 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:35.083+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:35.448 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:35.083+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:35.448 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:35.083+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:35.448 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:35.084+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:35.778 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:35.552+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:35.778 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:35.552+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:35.778 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:35.552+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:35.778 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:35.552+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:36.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:35.778+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:36.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:35.778+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:36.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:35.778+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:36.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:35 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:35.779+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:36.464 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:25:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:25:36.056+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 35725 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:25:36.465 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:36.034+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:36.465 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:36.034+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:36.465 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:36.034+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:36.465 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:36.035+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:36.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:36.514+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:36.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:36.514+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:36.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:36.514+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:36.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:36.515+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:37.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:37.037+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:37.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:37.037+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:37.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:37.037+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:37.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:37.037+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:37.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:36.817+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:37.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:36.817+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:37.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:36.817+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:37.283 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:36 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:36.818+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:38.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:37.774+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:38.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:37.774+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:38.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:37.774+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:38.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:37.775+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:38.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:37.561+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:38.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:37.561+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:38.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:37.561+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:38.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:37 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:37.562+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:38.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:38.054+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:38.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:38.054+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:38.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:38.054+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:38.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:38.055+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:38.808+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:38.808+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:39.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:38.808+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:39.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:38.808+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:39.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:38.587+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:39.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:38.587+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:39.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:38.587+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:39.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:38 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:38.587+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:39.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:39.045+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:39.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:39.045+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:39.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:39.045+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:39.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:39.046+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:39.842 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:39.565+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:39.843 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:39.565+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:39.843 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:39.565+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:39.843 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:39.565+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:40.225 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:40.068+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:40.226 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:40.068+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:40.226 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:40.068+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:40.226 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:40.069+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:40.226 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:39.842+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:40.226 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:39.842+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:40.226 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:39.842+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:40.227 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:39 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:39.843+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:40.323 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:25:40.324 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:25:40.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:40.607+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:40.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:40.607+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:40.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:40.607+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:40.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:40.608+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:41.060 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:40.808+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:41.061 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:40.808+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:41.061 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:40.808+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:41.061 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:40 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:40.809+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:41.061 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:41.049+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:41.061 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:41.049+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:41.062 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:41.049+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:41.062 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:41.049+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:41.532 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:25:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:25:41.060+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 35814 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:25:42.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:41.812+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:42.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:41.812+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:42.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:41.812+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:42.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:41.813+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:42.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:41.651+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:42.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:41.651+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:42.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:41.651+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:42.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:41 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:41.652+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:42.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:42.086+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:42.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:42.086+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:42.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:42.086+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:42.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:42.086+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:43.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:42.613+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:43.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:42.613+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:43.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:42.613+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:43.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:42.614+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:43.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:42.850+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:43.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:42.850+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:43.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:42.850+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:43.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:42 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:42.851+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:43.390 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:25:43.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:43.132+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:43.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:43.132+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:43.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:43.132+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:43.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:43.133+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:44.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:43.634+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:44.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:43.634+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:44.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:43.634+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:44.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:43.635+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:44.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:43.808+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:44.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:43.808+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:44.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:43.808+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:44.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:43 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:43.809+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:44.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:44.137+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:44.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:44.137+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:44.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:44.137+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:44.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:44.137+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:45.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:44.587+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:45.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:44.587+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:45.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:44.587+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:45.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:44.588+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:45.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:44.773+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:45.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:44.773+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:45.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:44.773+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:45.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:44 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:44.774+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:45.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:45.145+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:45.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:45.145+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:45.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:45.145+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:45.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:45.146+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:46.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:45.551+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:46.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:45.551+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:46.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:45.551+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:46.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:45.552+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:46.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:45.781+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:46.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:45.781+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:46.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:45.781+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:46.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:45 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:45.781+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:46.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:25:46.392 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:25:46.466 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:25:46.466 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-24T10:25:46.467 INFO:tasks.cephadm:Removing cluster... 2024-03-24T10:25:46.467 DEBUG:teuthology.orchestra.run.smithi022:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 45356448-e9c2-11ee-95cd-87774f69a715 --force 2024-03-24T10:25:46.490 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:25:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:25:46.065+0000 7f6a48ee5700 -1 mon.smithi022@0(probing) e2 get_health_metrics reporting 35904 slow ops, oldest is log(2 entries from seq 291 at 2024-03-24T09:48:24.436205+0000) 2024-03-24T10:25:46.491 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:46.099+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:46.491 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:46.099+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:46.491 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:46.099+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:46.491 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:46.100+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:46.615 INFO:teuthology.orchestra.run.smithi022.stdout:Deleting cluster with fsid: 45356448-e9c2-11ee-95cd-87774f69a715 2024-03-24T10:25:46.756 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:25:46 smithi022 systemd[1]: Stopping Ceph mon.smithi022 for 45356448-e9c2-11ee-95cd-87774f69a715... 2024-03-24T10:25:46.756 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:46.560+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:46.756 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:46.560+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:46.756 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:46.560+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:46.756 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:46.561+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:47.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:46.755+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:47.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:46.755+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:47.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:46.755+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:47.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:46 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:46.756+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:47.441 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:25:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:25:47.085+0000 7f6a4beeb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi022 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T10:25:47.441 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:25:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022[29508]: 2024-03-24T10:25:47.085+0000 7f6a4beeb700 -1 mon.smithi022@0(probing) e2 *** Got Signal Terminated *** 2024-03-24T10:25:47.441 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:47.131+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:47.441 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:47.131+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:47.441 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:47.131+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:47.442 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:47.132+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:47.775 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:25:47 smithi022 podman[87366]: 2024-03-24 10:25:47.441186567 +0000 UTC m=+0.586828744 container died 59736938121b1d44b215afbc87b4b244b17d77969e70b1ae08d6bab98c5642fd (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022, io.buildah.version=1.33.5, RELEASE=reef-738c2fd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-24T10:25:47.776 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:47.549+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:47.776 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:47.549+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:47.776 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:47.549+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:47.776 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:47.550+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:48.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:25:47 smithi022 podman[87366]: 2024-03-24 10:25:47.986453036 +0000 UTC m=+1.132095212 container cleanup 59736938121b1d44b215afbc87b4b244b17d77969e70b1ae08d6bab98c5642fd (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=reef-738c2fd, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-24T10:25:48.032 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:25:47 smithi022 bash[87366]: ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022 2024-03-24T10:25:48.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:47.775+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:48.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:47.775+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:48.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:47.775+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:48.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:47 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:47.776+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:48.451 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:25:48 smithi022 podman[87378]: 2024-03-24 10:25:48.229804384 +0000 UTC m=+0.786426304 container remove 59736938121b1d44b215afbc87b4b244b17d77969e70b1ae08d6bab98c5642fd (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-45356448-e9c2-11ee-95cd-87774f69a715-mon-smithi022, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=reef-738c2fd, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319) 2024-03-24T10:25:48.451 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:48.091+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:48.452 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:48.091+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:48.452 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:48.091+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:48.452 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:48.092+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:48.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:25:48 smithi022 systemd[1]: ceph-45356448-e9c2-11ee-95cd-87774f69a715@mon.smithi022.service: Deactivated successfully. 2024-03-24T10:25:48.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:25:48 smithi022 systemd[1]: Stopped Ceph mon.smithi022 for 45356448-e9c2-11ee-95cd-87774f69a715. 2024-03-24T10:25:48.782 INFO:journalctl@ceph.mon.smithi022.smithi022.stdout:Mar 24 10:25:48 smithi022 systemd[1]: ceph-45356448-e9c2-11ee-95cd-87774f69a715@mon.smithi022.service: Consumed 26.123s CPU time. 2024-03-24T10:25:48.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:48.570+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:48.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:48.570+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:48.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:48.570+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:48.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:48.571+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:49.050 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:48.788+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:49.051 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:48.788+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:49.051 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:48.788+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:49.051 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:48 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:48.789+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:49.358 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:49.050+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:49.358 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:49.050+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:49.358 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:49.050+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:49.359 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:49.051+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:49.668 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:49.540+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:49.668 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:49.540+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:49.668 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:49.540+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:49.668 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:49.541+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:50.022 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:49.752+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:50.023 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:49.752+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:50.023 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:49.752+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:50.023 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:49 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:49.753+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:50.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:50.021+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:50.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:50.021+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:50.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:50.021+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:50.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:50.022+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:50.756 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:50.549+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:50.756 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:50.549+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:50.756 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:50.549+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:50.757 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:50.550+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:51.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:50.772+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:51.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:50.772+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:51.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:50.772+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:51.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:50 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:50.773+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:51.325 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:51.036+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:51.325 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:51.036+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:51.325 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:51.036+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:51.326 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:51.037+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:51.847 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:51.819+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:51.847 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:51.819+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:51.847 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:51.819+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:51.847 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:51.820+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:51.848 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:51.588+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:51.848 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:51.588+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:51.848 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:51.588+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:51.848 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:51 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:51.588+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:52.145 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:52.033+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:52.145 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:52.033+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:52.145 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:52.033+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:52.145 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:52.034+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:52.759 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:52.599+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:52.760 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:52.599+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:52.760 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:52.599+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:52.760 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:52.600+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:53.015 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:52.774+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:53.015 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:52.774+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:53.015 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:52.774+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:53.016 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:52 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:52.775+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:53.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:53.057+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:53.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:53.057+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:53.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:53.057+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:53.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:53.058+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:53.950 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:53.802+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:53.950 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:53.802+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:53.950 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:53.802+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:53.951 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:53.803+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:53.951 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:53.648+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:53.951 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:53.648+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:53.951 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:53.648+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:53.951 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:53 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:53.649+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:54.281 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:54.036+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:54.281 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:54.036+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:54.281 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:54.036+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:54.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:54.037+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:54.913 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:54.695+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:54.914 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:54.695+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:54.914 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:54.695+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:54.914 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:54.696+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:54.914 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:54.767+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:54.915 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:54.767+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:54.915 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:54.767+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:54.915 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:54 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:54.767+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:55.247 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:55.032+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:55.248 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:55.032+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:55.248 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:55.032+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:55.248 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:55.033+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:56.018 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:55.746+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:56.018 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:55.746+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:56.018 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:55.746+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:56.018 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:55.746+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:56.019 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:55.734+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:56.019 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:55.734+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:56.019 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:55.734+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:56.019 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:55 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:55.735+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:56.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:56.017+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:56.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:56.017+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:56.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:56.017+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:56.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:56.018+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:57.025 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:56.777+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:57.025 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:56.777+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:57.025 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:56.777+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:57.026 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:56.777+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:57.026 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:56.699+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:57.026 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:56.699+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:57.026 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:56.699+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:57.026 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:56 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:56.701+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:57.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:57.024+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:57.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:57.024+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:57.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:57.024+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:57.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:57.025+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:57.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:57.770+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:57.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:57.770+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:57.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:57.770+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:57.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:57.770+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:57.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:57.669+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:57.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:57.669+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:57.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:57.669+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:57.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:57 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:57.670+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:58.249 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:58.060+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:58.250 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:58.060+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:58.250 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:58.060+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:58.250 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:58.062+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:58.853 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:58 smithi022 systemd[1]: Stopping Ceph osd.0 for 45356448-e9c2-11ee-95cd-87774f69a715... 2024-03-24T10:25:58.853 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:58.630+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:58.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:58.630+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:58.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:58.630+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:25:58.854 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:58.631+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:58.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:58.744+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:58.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:58.744+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:58.854 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:58.744+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:25:58.855 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:58.745+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:25:59.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:58.852+0000 7f897cd58700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T10:25:59.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:58.852+0000 7f897cd58700 -1 osd.0 46 *** Got signal Terminated *** 2024-03-24T10:25:59.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:58 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:58.852+0000 7f897cd58700 -1 osd.0 46 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T10:25:59.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:59.051+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:59.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:59.051+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:59.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:59.051+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:25:59.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:25:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:25:59.052+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:00.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:59.741+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:26:00.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:59.741+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:26:00.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:59.741+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:26:00.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:25:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:25:59.741+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:00.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:59.659+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:00.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:59.659+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:00.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:59.659+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:00.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:25:59 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:25:59.660+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:00.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:26:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:26:00.093+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:26:00.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:26:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:26:00.093+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:26:00.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:26:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:26:00.093+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:26:00.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:26:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:26:00.094+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:01.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:00.750+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:26:01.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:00.750+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:26:01.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:00.750+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:26:01.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:00.752+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:01.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:00.687+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:01.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:00.687+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:01.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:00.687+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:01.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:00 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:00.688+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:01.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:26:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:26:01.062+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:26:01.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:26:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:26:01.062+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:26:01.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:26:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:26:01.062+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:26:01.533 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:26:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:26:01.063+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:02.027 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:01.733+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:26:02.027 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:01.733+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:26:02.028 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:01.733+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:26:02.028 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:01.733+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:02.028 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:01.644+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:02.028 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:01.644+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:02.028 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:01.644+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:02.028 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:01 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:01.645+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:02.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:26:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:26:02.026+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:26:02.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:26:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:26:02.026+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:26:02.282 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:26:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:26:02.026+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:26:02.283 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:26:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:26:02.028+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:03.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:02.733+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:26:03.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:02.733+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:26:03.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:02.733+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:26:03.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:02.734+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:03.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:02.642+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:03.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:02.642+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:03.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:02.642+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:03.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:02 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:02.642+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:03.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:26:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:26:03.043+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:26:03.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:26:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:26:03.043+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:26:03.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:26:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:26:03.043+0000 7f8979b4f700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:19:59.989815+0000 (oldest deadline 2024-03-24T10:20:19.989815+0000) 2024-03-24T10:26:03.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:26:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0[48810]: 2024-03-24T10:26:03.044+0000 7f8979b4f700 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5708 3.6 3:66acd5c2:::10000004468.00000000:head [write 0~1320 in=1320b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:04.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:03.759+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:26:04.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:03.759+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:26:04.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:03.759+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:26:04.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:03.760+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:04.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:03.648+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:04.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:03.648+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:04.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:03.648+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:04.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:03 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:03.649+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:04.724 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:26:04 smithi022 podman[88327]: 2024-03-24 10:26:04.456957981 +0000 UTC m=+5.730115404 container died db726da62a1cc727d000ec4cf3256dd7cfed787a338dad836a62db246cd718d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-738c2fd, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-24T10:26:04.725 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:04.623+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:04.725 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:04.623+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:04.725 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:04.623+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:04.725 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:04.623+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:05.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:26:04 smithi022 podman[88327]: 2024-03-24 10:26:04.854379344 +0000 UTC m=+6.127536754 container cleanup db726da62a1cc727d000ec4cf3256dd7cfed787a338dad836a62db246cd718d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-738c2fd, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-24T10:26:05.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:26:04 smithi022 bash[88327]: ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0 2024-03-24T10:26:05.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:04.724+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:26:05.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:04.724+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:26:05.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:04.724+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:26:05.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:04 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:04.725+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:05.532 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:26:05 smithi022 podman[88345]: 2024-03-24 10:26:05.23918392 +0000 UTC m=+0.784531528 container remove db726da62a1cc727d000ec4cf3256dd7cfed787a338dad836a62db246cd718d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-738c2fd, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-24T10:26:05.917 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:05.724+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:26:05.918 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:05.724+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:26:05.918 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:05.724+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:26:05.918 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:05.724+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:05.919 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:05.635+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:05.919 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:05.635+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:05.919 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:05.635+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:05.919 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:05 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:05.635+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:05.920 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:26:05 smithi022 podman[88444]: 2024-03-24 10:26:05.532630619 +0000 UTC m=+0.018831018 image pull 68aff9aff3b2fe3c16548acc72564779ed9f377b279f3076ce38885b01df0d46 quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d 2024-03-24T10:26:05.920 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:26:05 smithi022 podman[88444]: 2024-03-24 10:26:05.665167478 +0000 UTC m=+0.151367875 container create 40b5a0c0566bec8991a504487ab57fffde7b0cb6e0e26ace9a2a60b0ed7f50bc (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=reef-738c2fd, CEPH_POINT_RELEASE=) 2024-03-24T10:26:05.920 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:26:05 smithi022 podman[88444]: 2024-03-24 10:26:05.917928323 +0000 UTC m=+0.404128735 container init 40b5a0c0566bec8991a504487ab57fffde7b0cb6e0e26ace9a2a60b0ed7f50bc (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, RELEASE=reef-738c2fd, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD) 2024-03-24T10:26:06.216 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:26:05 smithi022 podman[88444]: 2024-03-24 10:26:05.921620257 +0000 UTC m=+0.407820654 container start 40b5a0c0566bec8991a504487ab57fffde7b0cb6e0e26ace9a2a60b0ed7f50bc (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=reef-738c2fd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-24T10:26:06.216 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:26:06 smithi022 podman[88444]: 2024-03-24 10:26:06.017911488 +0000 UTC m=+0.504111900 container attach 40b5a0c0566bec8991a504487ab57fffde7b0cb6e0e26ace9a2a60b0ed7f50bc (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=reef-738c2fd, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-24T10:26:06.497 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:26:06 smithi022 podman[88444]: 2024-03-24 10:26:06.217580455 +0000 UTC m=+0.703780852 container died 40b5a0c0566bec8991a504487ab57fffde7b0cb6e0e26ace9a2a60b0ed7f50bc (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=reef-738c2fd, org.label-schema.vendor=CentOS) 2024-03-24T10:26:06.749 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:06.603+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:06.749 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:06.603+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:06.749 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:06.603+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:06.750 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:06.604+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:07.032 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:26:06 smithi022 podman[88470]: 2024-03-24 10:26:06.888937718 +0000 UTC m=+0.666789640 container remove 40b5a0c0566bec8991a504487ab57fffde7b0cb6e0e26ace9a2a60b0ed7f50bc (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-0-deactivate, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, RELEASE=reef-738c2fd, org.label-schema.vendor=CentOS) 2024-03-24T10:26:07.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:26:06 smithi022 systemd[1]: ceph-45356448-e9c2-11ee-95cd-87774f69a715@osd.0.service: Deactivated successfully. 2024-03-24T10:26:07.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:26:06 smithi022 systemd[1]: Stopped Ceph osd.0 for 45356448-e9c2-11ee-95cd-87774f69a715. 2024-03-24T10:26:07.033 INFO:journalctl@ceph.osd.0.smithi022.stdout:Mar 24 10:26:06 smithi022 systemd[1]: ceph-45356448-e9c2-11ee-95cd-87774f69a715@osd.0.service: Consumed 18min 21.949s CPU time. 2024-03-24T10:26:07.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:06.748+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:26:07.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:06.748+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:26:07.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:06.748+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:26:07.034 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:06 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:06.749+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:07.400 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:07 smithi022 systemd[1]: Stopping Ceph osd.1 for 45356448-e9c2-11ee-95cd-87774f69a715... 2024-03-24T10:26:07.723 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:07.585+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:07.724 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:07.585+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:07.724 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:07.585+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:07.724 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:07.586+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:07.724 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:07.399+0000 7fcd31c7a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T10:26:07.724 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:07.399+0000 7fcd31c7a700 -1 osd.1 46 *** Got signal Terminated *** 2024-03-24T10:26:07.725 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:07.399+0000 7fcd31c7a700 -1 osd.1 46 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T10:26:07.725 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:07.723+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:26:07.725 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:07.723+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:26:07.725 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:07.723+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:26:08.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:07 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:07.724+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:09.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:08.725+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:26:09.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:08.725+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:26:09.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:08.725+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:26:09.033 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:08.726+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:09.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:08.550+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:09.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:08.550+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:09.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:08.550+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:09.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:08 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:08.550+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:09.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:09.505+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:09.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:09.505+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:09.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:09.505+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:09.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:09.506+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:09.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:09.712+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:26:09.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:09.712+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:26:09.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:09.712+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:26:09.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:09 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:09.713+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:10.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:10.468+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:10.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:10.468+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:10.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:10.468+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:10.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:10.469+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:10.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:10.695+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:26:10.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:10.695+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:26:10.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:10.695+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:26:10.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:10 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:10.696+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:11.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:11.671+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:26:11.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:11.671+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:26:11.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:11.671+0000 7fcd2da6f700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:05.999782+0000 (oldest deadline 2024-03-24T10:20:25.999782+0000) 2024-03-24T10:26:11.783 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1[56507]: 2024-03-24T10:26:11.672+0000 7fcd2da6f700 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24303.0:5704 3.19 3:9e1c4d46:::10000004464.00000000:head [write 0~1683 [1@-1] in=1683b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:11.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:11.440+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:11.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:11.440+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:11.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:11.440+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:11.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:11 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:11.441+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:12.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:12.490+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:12.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:12.490+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:12.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:12.490+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:12.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:12 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:12.491+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:13.267 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:12 smithi022 podman[88558]: 2024-03-24 10:26:12.923284177 +0000 UTC m=+5.646044915 container died 3c6af2f5b6f13b2030d3dc413704483afad8678d741d9728b0b10ab1b0af1d31 (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=reef-738c2fd, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-24T10:26:13.518 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:13 smithi022 podman[88558]: 2024-03-24 10:26:13.366765421 +0000 UTC m=+6.089526165 container cleanup 3c6af2f5b6f13b2030d3dc413704483afad8678d741d9728b0b10ab1b0af1d31 (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-738c2fd, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-24T10:26:13.519 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:13 smithi022 bash[88558]: ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1 2024-03-24T10:26:13.782 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:13 smithi022 podman[88580]: 2024-03-24 10:26:13.551223484 +0000 UTC m=+0.625608598 container remove 3c6af2f5b6f13b2030d3dc413704483afad8678d741d9728b0b10ab1b0af1d31 (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, RELEASE=reef-738c2fd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-24T10:26:13.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:13.518+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:13.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:13.518+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:13.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:13.518+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:13.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:13 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:13.518+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:14.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:13 smithi022 podman[88671]: 2024-03-24 10:26:13.878560532 +0000 UTC m=+0.018789030 image pull 68aff9aff3b2fe3c16548acc72564779ed9f377b279f3076ce38885b01df0d46 quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d 2024-03-24T10:26:14.282 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:14 smithi022 podman[88671]: 2024-03-24 10:26:14.077559336 +0000 UTC m=+0.217787832 container create 08ed8198c678e2687628a1a8676df2716d737ce2d45d38c4848aa7d505e99391 (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=reef-738c2fd, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-24T10:26:14.609 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:14.525+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:14.609 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:14.525+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:14.609 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:14.525+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:14.609 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:14 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:14.526+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:14.610 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:14 smithi022 podman[88671]: 2024-03-24 10:26:14.310141487 +0000 UTC m=+0.450370009 container init 08ed8198c678e2687628a1a8676df2716d737ce2d45d38c4848aa7d505e99391 (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-738c2fd) 2024-03-24T10:26:14.610 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:14 smithi022 podman[88671]: 2024-03-24 10:26:14.313172811 +0000 UTC m=+0.453401313 container start 08ed8198c678e2687628a1a8676df2716d737ce2d45d38c4848aa7d505e99391 (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=reef-738c2fd, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-24T10:26:14.610 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:14 smithi022 podman[88671]: 2024-03-24 10:26:14.39652468 +0000 UTC m=+0.536753195 container attach 08ed8198c678e2687628a1a8676df2716d737ce2d45d38c4848aa7d505e99391 (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=reef-738c2fd, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-24T10:26:15.032 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:14 smithi022 podman[88671]: 2024-03-24 10:26:14.609798589 +0000 UTC m=+0.750027086 container died 08ed8198c678e2687628a1a8676df2716d737ce2d45d38c4848aa7d505e99391 (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=reef-738c2fd, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T10:26:15.344 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:15 smithi022 podman[88697]: 2024-03-24 10:26:15.141513498 +0000 UTC m=+0.526317728 container remove 08ed8198c678e2687628a1a8676df2716d737ce2d45d38c4848aa7d505e99391 (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-1-deactivate, RELEASE=reef-738c2fd, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-24T10:26:15.344 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:15 smithi022 systemd[1]: ceph-45356448-e9c2-11ee-95cd-87774f69a715@osd.1.service: Deactivated successfully. 2024-03-24T10:26:15.344 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:15 smithi022 systemd[1]: Stopped Ceph osd.1 for 45356448-e9c2-11ee-95cd-87774f69a715. 2024-03-24T10:26:15.344 INFO:journalctl@ceph.osd.1.smithi022.stdout:Mar 24 10:26:15 smithi022 systemd[1]: ceph-45356448-e9c2-11ee-95cd-87774f69a715@osd.1.service: Consumed 18min 26.528s CPU time. 2024-03-24T10:26:15.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:15 smithi022 systemd[1]: Stopping Ceph osd.2 for 45356448-e9c2-11ee-95cd-87774f69a715... 2024-03-24T10:26:15.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:15.552+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:15.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:15.552+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:15.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:15.552+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:15.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:15.552+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:15.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:15.585+0000 7f65a36d5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T10:26:15.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:15.585+0000 7f65a36d5700 -1 osd.2 46 *** Got signal Terminated *** 2024-03-24T10:26:15.784 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:15 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:15.585+0000 7f65a36d5700 -1 osd.2 46 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T10:26:16.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:16.530+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:16.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:16.530+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:16.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:16.530+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:16.783 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:16 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:16.530+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:17.830 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:17.534+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:17.831 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:17.534+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:17.831 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:17.534+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:17.831 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:17 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:17.535+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:19.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:18.559+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:19.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:18.559+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:19.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:18.559+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:19.034 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:18 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:18.560+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:20.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:19.567+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:20.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:19.567+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:20.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:19.567+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:20.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:19 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:19.568+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:21.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:20.550+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:21.032 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:20.550+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:21.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:20.550+0000 7f65a04cc700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.155:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T10:20:10.041738+0000 (oldest deadline 2024-03-24T10:20:30.041738+0000) 2024-03-24T10:26:21.033 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:20 smithi022 ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2[64348]: 2024-03-24T10:26:20.550+0000 7f65a04cc700 -1 osd.2 46 get_health_metrics reporting 254 slow ops, oldest is osd_op(client.24303.0:5707 3.7 3:e45395c2:::10000004467.00000000:head [write 0~2578 [1@-1] in=2578b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T10:26:21.452 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:21 smithi022 podman[88775]: 2024-03-24 10:26:21.146819105 +0000 UTC m=+5.659987836 container died 99b3c8840e7d4e3c17f0278c1e178c7fa700d6ab118f2e3c62ceb3e28ed332b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=reef-738c2fd, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-24T10:26:21.452 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:21 smithi022 podman[88775]: 2024-03-24 10:26:21.452893851 +0000 UTC m=+5.966062586 container cleanup 99b3c8840e7d4e3c17f0278c1e178c7fa700d6ab118f2e3c62ceb3e28ed332b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=reef-738c2fd, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-24T10:26:21.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:21 smithi022 bash[88775]: ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2 2024-03-24T10:26:21.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:21 smithi022 podman[88804]: 2024-03-24 10:26:21.653928919 +0000 UTC m=+0.507640072 container remove 99b3c8840e7d4e3c17f0278c1e178c7fa700d6ab118f2e3c62ceb3e28ed332b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=reef-738c2fd, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-24T10:26:22.381 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:22 smithi022 podman[88898]: 2024-03-24 10:26:21.983863759 +0000 UTC m=+0.018968434 image pull 68aff9aff3b2fe3c16548acc72564779ed9f377b279f3076ce38885b01df0d46 quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d 2024-03-24T10:26:22.382 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:22 smithi022 podman[88898]: 2024-03-24 10:26:22.096341609 +0000 UTC m=+0.131446282 container create 48762aa657002079dc1b3c39604b51750aefbcf14af276cfa38052c6131188fb (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, RELEASE=reef-738c2fd, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-24T10:26:22.382 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:22 smithi022 podman[88898]: 2024-03-24 10:26:22.290229998 +0000 UTC m=+0.325334684 container init 48762aa657002079dc1b3c39604b51750aefbcf14af276cfa38052c6131188fb (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=reef-738c2fd, ceph=True, org.label-schema.vendor=CentOS) 2024-03-24T10:26:22.382 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:22 smithi022 podman[88898]: 2024-03-24 10:26:22.293166317 +0000 UTC m=+0.328270991 container start 48762aa657002079dc1b3c39604b51750aefbcf14af276cfa38052c6131188fb (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=reef-738c2fd, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS) 2024-03-24T10:26:22.382 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:22 smithi022 podman[88898]: 2024-03-24 10:26:22.381930723 +0000 UTC m=+0.417035421 container attach 48762aa657002079dc1b3c39604b51750aefbcf14af276cfa38052c6131188fb (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=reef-738c2fd, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-24T10:26:22.782 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:22 smithi022 podman[88898]: 2024-03-24 10:26:22.593335388 +0000 UTC m=+0.628440054 container died 48762aa657002079dc1b3c39604b51750aefbcf14af276cfa38052c6131188fb (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=reef-738c2fd, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T10:26:23.126 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:23 smithi022 podman[88924]: 2024-03-24 10:26:23.110413327 +0000 UTC m=+0.512230990 container remove 48762aa657002079dc1b3c39604b51750aefbcf14af276cfa38052c6131188fb (image=quay.ceph.io/ceph-ci/ceph@sha256:98326f2c709ec05ace1337a0445b7e81810fa2d7810e897bde71d5d14691565d, name=ceph-45356448-e9c2-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=reef-738c2fd, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T10:26:23.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:23 smithi022 systemd[1]: ceph-45356448-e9c2-11ee-95cd-87774f69a715@osd.2.service: Deactivated successfully. 2024-03-24T10:26:23.532 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:23 smithi022 systemd[1]: Stopped Ceph osd.2 for 45356448-e9c2-11ee-95cd-87774f69a715. 2024-03-24T10:26:23.533 INFO:journalctl@ceph.osd.2.smithi022.stdout:Mar 24 10:26:23 smithi022 systemd[1]: ceph-45356448-e9c2-11ee-95cd-87774f69a715@osd.2.service: Consumed 24.466s CPU time. 2024-03-24T10:26:27.835 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:26:27.837 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:26:30.914 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:26:39.916 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:26:39.916 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:26:42.978 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:26:54.979 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:26:54.980 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:26:58.046 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:27:01.048 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:27:01.048 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:27:01.118 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:27:01.118 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-24T10:27:01.119 INFO:tasks.cephadm:Teardown complete 2024-03-24T10:27:01.119 ERROR:teuthology.run_tasks:Manager failed: cephadm Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephadm.py", line 1553, in distribute_config_and_admin_keyring yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephadm.py", line 1807, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephadm.py", line 1810, in task log.info('Teardown begin') File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephadm.py", line 1686, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephadm.py", line 142, in download_cephadm _rm_cluster(ctx, cluster_name) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephadm.py", line 310, in _rm_cluster ctx.cluster.run(args=[ File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi155 2024-03-24T10:27:01.149 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-24T10:27:01.163 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-03-24T10:27:01.180 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-24T10:27:01.180 DEBUG:teuthology.orchestra.run.smithi022:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-24T10:27:01.192 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:27:01.192 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:27:04.258 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:27:13.260 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:27:13.261 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:27:16.318 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:27:28.319 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:27:28.321 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:27:31.390 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:27:34.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:27:34.392 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:27:34.466 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:27:34.466 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-24T10:27:34.510 ERROR:teuthology.run_tasks:Manager failed: install Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/util.py", line 148, in ship_utilities yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 626, in task yield File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 220, in install yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/util.py", line 150, in ship_utilities _remove_utilities(ctx, filenames) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/util.py", line 121, in _remove_utilities ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi155 2024-03-24T10:27:34.534 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-24T10:27:34.549 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-24T10:27:34.550 DEBUG:teuthology.orchestra.run.smithi022:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T10:27:34.558 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:27:34.558 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:27:34.572 INFO:teuthology.orchestra.run.smithi022.stderr:bash: line 1: ntpq: command not found 2024-03-24T10:27:34.580 INFO:teuthology.orchestra.run.smithi022.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T10:27:34.580 INFO:teuthology.orchestra.run.smithi022.stdout:=============================================================================== 2024-03-24T10:27:34.580 INFO:teuthology.orchestra.run.smithi022.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 43 +136us[ +137us] +/- 85ms 2024-03-24T10:27:34.580 INFO:teuthology.orchestra.run.smithi022.stdout:^* hv02.front.sepia.ceph.com 3 6 377 42 -615us[ -614us] +/- 60ms 2024-03-24T10:27:34.580 INFO:teuthology.orchestra.run.smithi022.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 44 +640us[ +641us] +/- 67ms 2024-03-24T10:27:34.580 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:27:37.630 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:27:46.631 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:27:46.631 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:27:49.698 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:28:01.699 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:28:01.701 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:28:04.766 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:28:07.768 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:28:07.769 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:28:07.838 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:28:07.839 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-24T10:28:07.839 ERROR:teuthology.run_tasks:Manager failed: clock Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/clock.py", line 66, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/clock.py", line 72, in task cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi155 2024-03-24T10:28:07.857 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-24T10:28:07.873 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-24T10:28:07.874 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-24T10:28:07.914 DEBUG:teuthology.orchestra.run.smithi022:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T10:28:08.004 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:28:08.005 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:28:11.070 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:28:20.071 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:28:20.073 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:28:23.138 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:28:35.139 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:28:35.140 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:28:38.210 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:28:41.211 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:28:41.212 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:28:41.278 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:28:41.278 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-24T10:28:41.279 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/selinux.py", line 165, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/selinux.py", line 195, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi155 2024-03-24T10:28:41.288 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-24T10:28:41.301 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-24T10:28:41.332 INFO:teuthology.task.internal:Duration was 3402.316066 seconds 2024-03-24T10:28:41.333 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-24T10:28:41.365 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-24T10:28:41.366 DEBUG:teuthology.orchestra.run.smithi022:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T10:28:41.371 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:28:41.372 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:28:41.419 INFO:teuthology.orchestra.run.smithi022.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T10:28:44.450 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:28:53.451 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:28:53.524 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:28:56.574 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:29:08.576 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:29:08.578 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:29:11.650 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:29:14.652 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:29:14.652 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:29:14.718 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:29:14.718 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-24T10:29:14.719 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/syslog.py", line 76, in syslog yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/syslog.py", line 85, in syslog cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi155 2024-03-24T10:29:14.737 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-24T10:29:14.751 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-24T10:29:14.751 DEBUG:teuthology.orchestra.run.smithi022:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T10:29:14.789 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:29:14.789 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:29:17.858 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:29:26.859 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:29:26.860 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:29:29.922 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:29:41.923 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:29:41.924 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:29:44.990 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:29:47.992 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:29:47.993 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:29:48.062 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:29:48.063 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-24T10:29:48.063 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 452, in sudo yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 455, in sudo ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi155 2024-03-24T10:29:48.082 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-24T10:29:48.100 DEBUG:teuthology.orchestra.run.smithi022:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T10:29:48.108 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:29:48.109 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:29:48.136 INFO:teuthology.orchestra.run.smithi022.stdout:kernel.core_pattern = core 2024-03-24T10:29:51.170 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:30:00.171 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:30:00.172 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:30:03.230 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:30:15.232 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:30:15.233 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:30:18.306 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:30:21.307 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:30:21.308 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:30:21.378 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:30:21.378 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-24T10:30:21.379 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 488, in coredump yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 492, in coredump cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi155 2024-03-24T10:30:21.396 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-24T10:30:21.410 INFO:teuthology.task.internal:Transferring archived files... 2024-03-24T10:30:21.411 DEBUG:teuthology.misc:Transferring archived files from smithi022:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619043/remote/smithi022 2024-03-24T10:30:21.411 DEBUG:teuthology.orchestra.run.smithi022:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T10:30:21.456 DEBUG:teuthology.misc:Transferring archived files from smithi155:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619043/remote/smithi155 2024-03-24T10:30:21.456 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:30:21.457 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:30:24.514 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:30:33.515 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:30:33.517 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:30:36.579 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:30:48.579 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:30:48.580 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:30:51.650 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:30:54.651 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:30:54.652 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:30:54.718 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:30:54.718 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-24T10:30:54.719 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 418, in archive misc.pull_directory(rem, archive_dir, path, maybe_compress) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 748, in pull_directory r = remote.get_tar_stream(remotedir, sudo=True, compress=False) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 678, in get_tar_stream return self.run(args=args, wait=False, stdout=run.PIPE) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi155 2024-03-24T10:30:54.736 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-24T10:30:54.750 INFO:teuthology.task.internal:Not uploading archives. 2024-03-24T10:30:54.751 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-24T10:30:54.763 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-24T10:30:54.763 DEBUG:teuthology.orchestra.run.smithi022:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T10:30:54.770 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:30:54.771 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:30:54.786 INFO:teuthology.orchestra.run.smithi022.stdout: 262166 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 24 10:27 /home/ubuntu/cephtest 2024-03-24T10:30:54.786 INFO:teuthology.orchestra.run.smithi022.stdout: 262167 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 24 10:29 /home/ubuntu/cephtest/archive 2024-03-24T10:30:54.786 INFO:teuthology.orchestra.run.smithi022.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 09:31 /home/ubuntu/cephtest/archive/syslog 2024-03-24T10:30:54.786 INFO:teuthology.orchestra.run.smithi022.stdout: 265306 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Mar 24 09:31 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T10:30:54.786 INFO:teuthology.orchestra.run.smithi022.stdout: 265305 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Mar 24 09:31 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T10:30:54.786 INFO:teuthology.orchestra.run.smithi022.stdout: 413336 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 10:28 /home/ubuntu/cephtest/archive/audit 2024-03-24T10:30:54.787 INFO:teuthology.orchestra.run.smithi022.stdout: 413338 180 -rw------- 1 ubuntu ubuntu 180783 Mar 24 10:28 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-03-24T10:30:54.788 INFO:teuthology.orchestra.run.smithi022.stdout: 276586 212 -rwxr-xr-x 1 ubuntu ubuntu 215105 Mar 24 09:38 /home/ubuntu/cephtest/cephadm 2024-03-24T10:30:54.789 INFO:teuthology.orchestra.run.smithi022.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-03-24T10:30:57.827 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:31:06.828 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:31:06.830 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:31:09.890 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:31:21.891 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:31:21.892 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:31:24.958 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:31:27.961 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-24T10:31:27.962 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:31:28.030 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-24T10:31:28.030 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-24T10:31:28.031 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 54, in base ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi155 2024-03-24T10:31:28.050 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-24T10:31:28.066 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-24T10:31:28.170 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: OSError: [Errno 113] No route to host 2024-03-24T10:31:28.170 INFO:teuthology.nuke:Checking targets against current locks 2024-03-24T10:31:28.207 DEBUG:teuthology.nuke:shortname: smithi022 2024-03-24T10:31:28.208 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T10:31:28.236 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi022.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619043', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 09:21:57.903781', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEZMU8wfzxXYjkToy1O0HCIJPcSvIjN9CZeeueWYCafZ3yHXFp7VdpukJH+++twxJdbF03+oDj2xsVwLpKYyyH4='} 2024-03-24T10:31:28.241 DEBUG:teuthology.nuke:shortname: smithi155 2024-03-24T10:31:28.241 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T10:31:28.266 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi155.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619043', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 09:21:57.902602', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEWIBKXao9l9HY/Ss+f6KvxgkCPZm0HzzlnoKyUtczCgpOtgmJicC3gx77Xry9WD0tK7rh+bATRAIlFjgHm5f/s='} 2024-03-24T10:31:28.337 INFO:teuthology.orchestra.console.smithi022:Power off 2024-03-24T10:31:28.338 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T10:31:28.393 DEBUG:teuthology.orchestra.console.smithi022:power off output: Chassis Power Control: Down/Off 2024-03-24T10:31:28.393 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T10:31:28.419 INFO:teuthology.orchestra.console.smithi155:Power off 2024-03-24T10:31:28.419 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T10:31:28.448 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is on 2024-03-24T10:31:28.479 DEBUG:teuthology.orchestra.console.smithi155:power off output: Chassis Power Control: Down/Off 2024-03-24T10:31:28.479 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T10:31:28.514 DEBUG:teuthology.orchestra.console.smithi155:check power output: Chassis Power is on 2024-03-24T10:31:32.450 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T10:31:32.515 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T10:31:32.563 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is on 2024-03-24T10:31:32.628 DEBUG:teuthology.orchestra.console.smithi155:check power output: Chassis Power is on 2024-03-24T10:31:36.568 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T10:31:36.628 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T10:31:36.683 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is off 2024-03-24T10:31:36.741 DEBUG:teuthology.orchestra.console.smithi155:check power output: Chassis Power is off 2024-03-24T10:31:36.785 INFO:teuthology.orchestra.console.smithi022:Power off completed 2024-03-24T10:31:36.843 INFO:teuthology.orchestra.console.smithi155:Power off completed 2024-03-24T10:31:36.949 INFO:teuthology.run:Summary data: description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 3402.316066265106 failure_reason: 'Command failed on smithi022 with status 1: ''sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45356448-e9c2-11ee-95cd-87774f69a715 -e sha1=21b3bd672163092b4e542f772797fcba02d3d4de -- bash -c \''ceph versions | jq -e \''"\''"\''.mgr | keys\''"\''"\'' | grep $sha1\''''' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=3ece1145024045678ceb6d1e7ef7e150 status: fail success: false 2024-03-24T10:31:36.949 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T10:31:37.049 INFO:teuthology.run:FAIL